tor-observe - tool to see to which destinations Tor is connecting to

[adrelanos]

Sometimes users request a feature to observe connections between Whonix-Workstation and Whonix-Gateway. But it's not really limited to Whonix. Also users that are using Tor Browser would like to check what connections Tor is actually creating. (related: #27)

In #13 (comment) a Tor Browser regression was found, Tor Browser (Firefox) phoning home to firefox.settings.services.mozilla.com:
https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/31575

Which you then reported at:
https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/40788

Including some potential other unwanted connections.

There might be many more such privacy violations. But these are difficult to spot when these are routed over Tor.

Therefore it would be useful to have some tool that users can run which shows everything where Tor is resolving DNS and connecting to.

• Output should be safe to copy/paste into public places such as mailing lists, forums, IRC without de-anonymizing oneself. Therefore Tor entry guard, middle relay and exit relay and even onion rendezvous relay should be omitted.
• Duplicates might often be not helpful, but sometimes helpful to see the how many there really are or guess what is causing it? Default probably best to not show duplicates?
• Simple to start monitoring by just staring 1 utility?
• Similar to tor-ctrl-stream but simpler and OK for copy/paste?
• If connecting to IPs, get the reverse DNS of the IP and add it to the script output to make it easier for the user to find out what kind of connection it is?
• Live. Show connections when they are created. Better usability than having to press another button to actually see what happened.
• Not sure if there is a way to find out which application was responsible for the traffic?

[nyxnor]

Therefore it would be useful to have some tool that users can run which shows everything where Tor is resolving DNS and connecting to.

Have you tested the new tor-ctrl-stream upgrades? I think this is exactly what you want:

tor-ctrl-stream -s 9151 -z

-z don't print circuit's path

It does not print circuits, it prints the hostname and ip.

[nyxnor]

Output should be safe to copy/paste into public places such as mailing lists, forums, IRC without de-anonymizing oneself. Therefore Tor entry guard, middle relay and exit relay and even onion rendezvous relay should be omitted.

Completed

Duplicates might often be not helpful, but sometimes helpful to see the how many there really are or guess what is causing it? Default probably best to not show duplicates?

I prefer duplicate so we can see how many times it was requested

Simple to start monitoring by just staring 1 utility?
Similar to tor-ctrl-stream but simpler and OK for copy/paste?
If connecting to IPs, get the reverse DNS of the IP and add it to the script output to make it easier for the user to find out what kind of connection it is?
Live. Show connections when they are created. Better usability than having to press another button to actually see what happened.

Complete

Not sure if there is a way to find out which application was responsible for the traffic?

When using torsocks, the torsocks appears on the SOCKS_USERNAME field. But besides that, I don't know how to determine which application made the request, which would be very useful to know.

[nyxnor]

Correction, no need to specify 9151 tcp socket as on whonix, TBB is using /run/tor/control.
tor-ctrl-stream -z
On TBB, connect to whonix.org.

StreamId StreamPurpose StreamTarget CircuitId CircuitPurpose
--------------------------------------------------------------------------------------------------------------
2964 USER whonix.org-([2a01:4f9:2a:1a05::2]:443) 4572  GENERAL
2965 USER whonix.org-([2a01:4f9:2a:1a05::2]:443) 4572  GENERAL
2966 USER whonix.org-([2a01:4f9:2a:1a05::2]:443) 4572  GENERAL

[adrelanos]

Output still contains some lines with $934 etc. Tor relay fingerprints. That should be hidden? Or better...

StreamId StreamPurpose StreamTarget CircuitId CircuitPurpose

Possible to hide everything that comes before with a more specialized command line parameter or this tool?

Yeah. Maybe tor-ctrl-stream does most things already. Then perhaps a wrapper script for simplicity that sets the parameters to show more relevant output for this only?

tor-ctrl-stream -z
tor-ctrl-stream: [info]: subscribed to Tor stream events, as soon as streams are created, output will be shown below.
tor-ctrl-stream: [warn]: posting these contents online can deanonymize the tor client.

In that case the warning would have to be removed or toned down?

tor-ctrl-stream -z
tor-ctrl-stream: [info]: subscribed to Tor stream events, as soon as streams are created, output will be shown below.
tor-ctrl-stream: [warn]: posting these contents online can deanonymize the tor client.
tor-ctrl: [info]: stop listening with Ctrl+C
250 OK
250 OK

Possible to hide the 250 OK?

DIR_FETCH - hide by default?

[nyxnor]

Output still contains some lines with $934 etc. Tor relay fingerprints. That should be hidden?
DIR_FETCH - hide by default?

Are those fingerprint lines related to DIR_FETCH, if yes I can use that to hide those lines.

Possible to hide the 250 OK?

Yes.

Possible to hide everything that comes before with a more specialized command line parameter or this tool?

It is possible, but you'd like to see the connections when streams are made yes, to be interactive during run time. For that I'd have to think more on how to do that, currently not possible.

Yeah. Maybe tor-ctrl-stream does most things already. Then perhaps a wrapper script for simplicity that sets the parameters to show more relevant output for this only?

You mean a specific field?

In that case the warning would have to be removed or toned down?

Indeed.

[adrelanos]

Output still contains some lines with $934 etc. Tor relay fingerprints. That should be hidden?
DIR_FETCH - hide by default?

Are those fingerprint lines related to DIR_FETCH, if yes I can use that to hide those lines.

Yes. Also:

650 STREAM 725 SENTCONNECT
650 STREAM 725 SUCCEEDED
650 STREAM 725 CLOSED

Possible to hide everything that comes before with a more specialized command line parameter or this tool?

Yeah. Maybe tor-ctrl-stream does most things already. Then perhaps a wrapper script for simplicity that sets the parameters to show more relevant output for this only?

You mean a specific field?

Just a wrapper tor-observe (better name) actually running tor-ctrl-circuit -z "$@".

[nyxnor]

Also:

650 STREAM 725 SENTCONNECT
650 STREAM 725 SUCCEEDED
650 STREAM 725 CLOSED

Didnt get it, do you have an output example? You want to cut that fields?

Just a wrapper tor-observe (better name) actually running tor-ctrl-circuit -z "$@".

Binding(merging) tor-ctrl-stream with tor-ctrl-circuit?

[adrelanos]

EDIT

[adrelanos]

Also:

650 STREAM 725 SENTCONNECT
650 STREAM 725 SUCCEEDED
650 STREAM 725 CLOSED

Didnt get it, do you have an output example? You want to cut that fields?

tor-ctrl-stream -z
tor-ctrl-stream: [info]: subscribed to Tor stream events, as soon as streams are created, output will be shown below.
tor-ctrl-stream: [warn]: posting these contents online can deanonymize the tor client.
tor-ctrl: [info]: stop listening with Ctrl+C
250 OK
250 OK
650 STREAM 725 NEW 0 x.x.x.x.$000.exit:443 PURPOSE=DIR_FETCH CLIENT_PROTOCOL=UNKNOWN NYM_EPOCH=0 SESSION_GROUP=-2 ISO_FIELDS=
650 STREAM 725 SENTCONNECT 659 x.x.x.x.$000.exit:443 CLIENT_PROTOCOL=UNKNOWN NYM_EPOCH=0 SESSION_GROUP=-2 ISO_FIELDS=
650 STREAM 725 SUCCEEDED 659 x.x.x.x.$000.exit:443 CLIENT_PROTOCOL=UNKNOWN NYM_EPOCH=0 SESSION_GROUP=-2 ISO_FIELDS=
650 STREAM 725 CLOSED 659 x.x.x.x.$000.exit:443 REASON=END REMOTE_REASON=DONE CLIENT_PROTOCOL=UNKNOWN NYM_EPOCH=0 SESSION_GROUP=-2 ISO_FIELDS=

The x.x.x.x are redacted IPs. The $000 are redacted relay fingerprints.

Just a wrapper tor-observe (better name) actually running tor-ctrl-circuit -z "$@".

Binding(merging) tor-ctrl-stream with tor-ctrl-circuit?

No.

I meant...

Just a wrapper tor-observe (better name) actually running tor-ctrl-stream -z "$@".

[nyxnor]

Not pushed to github yet

• remove DIR_FETCH from private tor-ctrl-stream, so not by default if not using -z.
• hide ^250
• tone down warning when private mode is enabled

Just a wrapper tor-observe (better name) actually running tor-ctrl-stream -z "$@".

I really don't get it. I will leave this for later, or if you want to contribute, I'd be glad.

[adrelanos]

Just a wrapper tor-observe (better name) actually running tor-ctrl-stream -z "$@".

I really don't get it. I will leave this for later, or if you want to contribute, I'd be glad.

Yeah. Not a big deal. And quicker done than explained. Will do later. :)

[nyxnor]

Not sure if there is a way to find out which application was responsible for the traffic?

650 DEBUG conn_read_callback(): socket 49 wants to read.
650 DEBUG connection_handle_listener_read(): Connection accepted on socket 183 (child of fd 49).
650 INFO connection_handle_listener_read(): New SOCKS connection opened from 10.X.X.10.
650 DEBUG connection_add_impl(): new conn type Socks, socket 183, address 10.X.X.10, n_conns 179.
650 DEBUG conn_read_callback(): socket 183 wants to read.
650 DEBUG read_to_chunk(): Read 3 bytes. 3 on inbuf.
650 DEBUG connection_ap_handshake_process_socks(): enter

Making a request on TorBrowser, enabled every event to find anything useful about the client, this is the maximum important information I got from the logs, the client ip 10.X.X.10 is the workstation.

[nyxnor]

The x.x.x.x are redacted IPs. The $000 are redacted relay fingerprints.

These were redacted from the table. Do you want the script to not show the raw logs of relays?

[adrelanos]

For something to be shared in public, better avoid?
(That's why I thought of for this ticket as a separate tool or parameter.)

Users will inevitably do a complete copy/paste of all outputs for such as "tor observe" tool (for lack of better term).

[nyxnor]

For something to be shared in public, better avoid?
(That's why I thought of for this ticket as a separate tool or parameter.)

Users will inevitably do a complete copy/paste of all outputs for such as "tor observe" tool (for lack of better term).

I get it now. tor-observe will be a wrapper for tor-ctrl-stream on private mode and hiding even more things. by default, so no need to specify options as people will forget about that.

[adrelanos]

Exactly.

[nyxnor]

650 STREAM 725 NEW 0 x.x.x.x.$000.exit:443 PURPOSE=DIR_FETCH CLIENT_PROTOCOL=UNKNOWN NYM_EPOCH=0 SESSION_GROUP=-2 ISO_FIELDS=
650 STREAM 725 SENTCONNECT 659 x.x.x.x.$000.exit:443 CLIENT_PROTOCOL=UNKNOWN NYM_EPOCH=0 SESSION_GROUP=-2 ISO_FIELDS=
650 STREAM 725 SUCCEEDED 659 x.x.x.x.$000.exit:443 CLIENT_PROTOCOL=UNKNOWN NYM_EPOCH=0 SESSION_GROUP=-2 ISO_FIELDS=
650 STREAM 725 CLOSED 659 x.x.x.x.$000.exit:443 REASON=END REMOTE_REASON=DONE CLIENT_PROTOCOL=UNKNOWN NYM_EPOCH=0 SESSION_GROUP=-2 ISO_FIELDS=

These lines are not printed on the table of tor-ctrl-stream when using -z.
Thinking about it, it just sees the purpose when opening the stream, so if the stream was already created before, it will still show the exit. So I need to exclude the pattern .exit:

[nyxnor]

• make tor-ctrl-observer

If I was lazy, I'd do

tor-ctrl -w -c "SETEVENTS STREAM" | sed "/DIR_FETCH/d;/\.exit\:/d;/^$/d" | \
cut -d " " -f3-6

but this wouldn't show StreamPurpose as it does not have a fixed field, could be on the 7th or 8th or not present at all if not a NEW event, and then it would show unwanted fields, SOCKS_USERNAME, SOCKS_PASSWORD, SOURCE_ADDR

These are the lines that will be printed:

1235 NEW 0 www.torproject.org:443
1234 NEW 0 www.torproject.org:443
1235 SENTCONNECT 997 www.torproject.org:443
1234 SENTCONNECT 997 www.torproject.org:443
1233 REMAP 997 [2a01:4f8:fff0:4f:266:37ff:fe2c:5d19]:443
1233 SUCCEEDED 997 [2a01:4f8:fff0:4f:266:37ff:fe2c:5d19]:443

If you think it is important for the user to know if the purpose was a DNS_REQUEST or USER, then I will need to improve a lot the parsing, using temporary files etc, just to get the purpose.

---

maye should hide these other DIR_* stream purposes?

   The "PURPOSE" field is provided only for NEW and NEWRESOLVE events, and
   only if extended events are enabled (see 3.19).  Clients MUST accept
   purposes not listed above.  The purposes above are defined as:

       "DIR_FETCH" -- This stream is generated internally to Tor for
         fetching directory information.
       "DIR_UPLOAD" -- An internal stream for uploading information to
         a directory authority.
       "DIRPORT_TEST" -- A stream we're using to test our own directory
         port to make sure it's reachable.
       "DNS_REQUEST" -- A user-initiated DNS request.
       "USER" -- This stream is handling user traffic, OR it's internal
         to Tor, but it doesn't match one of the purposes above.

[nyxnor]

tor-ctrl-observer finished:

^CTerminated

StreamId StreamPurpose StreamTarget CircuitId CircuitPurpose
--------------------------------------------------------------------------------------------------------------
2004 USER duckduckgo.com-(52.142.124.215:443) 1266 GENERAL
2005 USER www.torproject.org-(95.216.163.36:443) 1255 GENERAL
2006 USER www.torproject.org-(95.216.163.36:443) 1255 GENERAL
2007 USER www.torproject.org-(95.216.163.36:443) 1255 GENERAL
2008 USER www.torproject.org:443 1255 GENERAL
2009 USER www.torproject.org-(95.216.163.36:443) 1255 GENERAL
2010 USER www.torproject.org-(95.216.163.36:443) 1255 GENERAL
2011 USER www.torproject.org-(95.216.163.36:443) 1255 GENERAL
2012 USER www.torproject.org-(95.216.163.36:443) 1255 GENERAL
2013 USER www.torproject.org-(95.216.163.36:443) 1255 GENERAL
2014 USER www.torproject.org-(95.216.163.36:443) 1255 GENERAL
2015 USER www.torproject.org-(95.216.163.36:443) 1255 GENERAL
2016 USER 2gzyxa5ihm7nsggfxnu52rck2vv4rvmdlkiu3zzui5du4xyclen53wid.onion:80 1260 HS_CLIENT_REND
2017 USER 2gzyxa5ihm7nsggfxnu52rck2vv4rvmdlkiu3zzui5du4xyclen53wid.onion:80 1260 HS_CLIENT_REND
2018 USER 2gzyxa5ihm7nsggfxnu52rck2vv4rvmdlkiu3zzui5du4xyclen53wid.onion:80 1260 HS_CLIENT_REND
2019 USER 2gzyxa5ihm7nsggfxnu52rck2vv4rvmdlkiu3zzui5du4xyclen53wid.onion:80 1260 HS_CLIENT_REND
2020 USER 2gzyxa5ihm7nsggfxnu52rck2vv4rvmdlkiu3zzui5du4xyclen53wid.onion:80 1260 HS_CLIENT_REND

Stream 2008 not showing torproject ip address is not a bug, stream was closed before being remapped.

[nyxnor]

Closing this. If unsatisfied, something to improve or remove, let me know.

[nyxnor]

bda0df7

stopped the filters on tor-ctrl-stream, so exits shown by DIR_FETCH will be shown there. Every stream must be shown here for advanced users.

The tool for privacy is tor-ctrl-observer, ther eis is filtered.