TypeError

[eldraco]

In dev (311b1e5)

I should say that I modify the file Malcom/networking/netsniffer.py and changed the line
self.filter = "ip and not host 127.0.0.1 and not host %s %s" % (remote_addr, filter_ifaces)
for
self.filter = ""

So I can sniff my own traffic.

[MODEL] - (updated hostname ip1a-lb3-prd.iad.github.com)
[MODEL] - (updated hostname 1.0.0.127.in-addr.arpa)
[DEBUG] - Caught DNS question: 1.0.0.127.in-addr.arpa
[DEBUG] - [+] DNS replies caught (1 answers)
[DEBUG] - No relevant records in reply
[DEBUG] - [+] DNS replies caught (1 answers)
[DEBUG] - No relevant records in reply
Exception in thread Thread-2:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(_self.__args, *_self.__kwargs)
File "/home/user/dir/aplics/malcom/Malcom/networking/netsniffer.py", line 90, in run
self.pkts += self.sniff(stopper=self.stop_sniffing, filter=self.filter, prn=self.handlePacket, stopperTimeout=1)
File "/home/user/dir/aplics/malcom/Malcom/networking/netsniffer.py", line 513, in sniff
r = prn(p)
File "/home/user/dir/aplics/malcom/Malcom/networking/netsniffer.py", line 382, in handlePacket
new_elts, new_edges = self.checkHTTP(flow)
File "/home/user/dir/aplics/malcom/Malcom/networking/netsniffer.py", line 323, in checkHTTP
if url['value'] not in self.nodes_values:
TypeError: list indices must be integers, not str

[tomchop]

You'll have a problem by sniffing with an empty filter: Malcom generates traffic to send node information to the browser in real time. If localhost (or at least the IP which is loading the webpage) is not excluded from the traffic, then you'll get an infinite loop (traffic detected, send info to websocket. Info to websocket is detected as traffic sent, send info to websocket).

That being said, the bug seems to come from somewhere else. What kind of traffic are you generating?

[eldraco]

Hi Thomas! Thanks for the answer.

Well, I was sniffing my normal traffic, that means I browse some web pages,
maybe some ssh, not too much. But the bug occurred like 5 seconds after
starting, so it was not a lot of traffic. I can make some tests later and
send them to you.
Also, maybe we can add a filter to exclude the websockets information
sending only? Instead of excluding your own IP completely.
I really think that sniffing your own traffic may be a good idea for a lot
of people.

cheers
sebas

On Tue, Dec 10, 2013 at 1:14 PM, Thomas Chopitea
notifications@github.comwrote:

You'll have a problem by sniffing with an empty filter: Malcom generates
traffic to send node information to the browser in real time. If localhost
(or at least the IP which is loading the webpage) is not excluded from the
traffic, then you'll get an infinite loop (traffic detected, send info to
websocket. Info to websocket is detected as traffic sent, send info to
websocket).

That being said, the bug seems to come from somewhere else. What kind of
traffic are you generating?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/7#issuecomment-30220620
.

[tomchop]

The reason I asked for network traffic is that I tried to reproduce the bug limiting traffic to everything except my browser's IP, did a couple of curl's from Malcom's command line and couldn't reproduce the issue. If you can get your hands on a pcap that would reproduce the issue, that would be great :)

In my vision, the traffic originating from Malcom's VM as well as the one originating from the browser that visits it should be segregated as much as possible from the other analysis in the network (so as to not pollute the graph).
That being said, it's definitely a good idea to leave this choice to the user. I'll include a checkbox to ask if we should include local traffic or not.

[eldraco]

Hi Thomas!
Well today I try again the problem.

And these are the results:
1st: I modified the line
self.filter = "ip and not host 127.0.0.1 and not host %s %s" %
(remote_addr, filter_ifaces)
for
self.filter = ""
in Malcom/networking/netsniffer.py

Git version is
On branch dev (commit 311b1e5)

2nd: I start a new sniffing session with name "test" and no filter. (I
tried with eth0 and wlan0)

After less than 10 packets the malcom gives me the same error:

[DEBUG] - Received: {u'session_name': u'test', u'cmd': u'sniffupdate'}
[MODEL] - (added hostname 8.8.8.8.in-addr.arpa)
[DEBUG] - Caught DNS question: 8.8.8.8.in-addr.arpa
[DEBUG] - [+] DNS replies caught (1 answers)
[DEBUG] - No relevant records in reply
[DEBUG] - [+] DNS replies caught (1 answers)
[DEBUG] - No relevant records in reply
Exception in thread Thread-2:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(_self.__args, *_self.__kwargs)
File
"/home/user/dir/aplics/traffic-analisis/malcom/Malcom/networking/netsniffer.py",
line 91, in run
self.pkts += self.sniff(stopper=self.stop_sniffing, filter=self.filter,
prn=self.handlePacket, stopperTimeout=1)
File
"/home/user/dir/aplics/traffic-analisis/malcom/Malcom/networking/netsniffer.py",
line 514, in sniff
r = prn(p)
File
"/home/user/dir/aplics/traffic-analisis/malcom/Malcom/networking/netsniffer.py",
line 383, in handlePacket
new_elts, new_edges = self.checkHTTP(flow)
File
"/home/user/dir/aplics/traffic-analisis/malcom/Malcom/networking/netsniffer.py",
line 324, in checkHTTP
if url['value'] not in self.nodes_values:
TypeError: list indices must be integers, not str

BUT!!!
If I capture those same packets in a pcap file and try to create a new
sniffing session with that pcap file... it does NOT explode!
So it only explode when it is a live capture! I can not give you a pcap
file.

If you try to sniff your own traffic like that, don't you have this issue?

cheersss!
sebas

On Tue, Dec 10, 2013 at 1:38 PM, Thomas Chopitea
notifications@github.comwrote:

The reason I asked for network traffic is that I tried to reproduce the
bug limiting traffic to everything except my browser's IP, did a couple of
curl's from Malcom's command line and couldn't reproduce the issue. If you
can get your hands on a pcap that would reproduce the issue, that would be
great :)

In my vision, the traffic originating from Malcom's VM as well as the one
originating from the browser that visits it should be segregated as much as
possible from the other analysis in the network (so as to not pollute the
graph).
That being said, it's definitely a good idea to leave this choice to the
user. I'll include a checkbox to ask if we should include local traffic
or not.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/7#issuecomment-30222049
.

[tomchop]

I've tried reproducing the bug, to no avail.
I looked at the code, the bug seems to come from the fact that add_text (line 321) returns more than one element. It would be awesome if you could add a print http_elts right before that line so that we can see what it's trying to add. I smell regex issues here :)