Onion site not reachable

[brassy-endomorph]

What version of GlobaLeaks are you using?

GlobaLeaks version: 4.13.18
Database version: 66
OS: Ubuntu 22.04.3

What browser(s) are you seeing the problem on?

No response

What operating system(s) are you seeing the problem on?

Linux

Describe the issue

The onion site is down and has been for several weeks. The GL application talks to the Tor socket, so this appears to be an application issue. There are no logs of any sort, so I have no idea what the issue could be.

Brought this to your attention here since apparently the discussion board goes unanswered

Proposed solution

Well. Restating GL, Tor, and the entire server does nothing, so fuck if I know what the issue is. Probably the code. Maybe add some logging so we can debug ourselves and then also fix it.

[evilaliv3]

Thank you @brassy-endomorph

I've still not answered as there is still no proper understanding of the failure or reproducibility but we are on it.

Do you have suggestions?

[evilaliv3]

@meejah: Apparently since we started spawning Tor via txttorcon time to time the ephemeral hidden services seems to not propagate correctly and users visiting the onion service receive the error " "Onion Site Not Found - Details: 0xF0 — The requested Onion service descriptor cannot't be found in the hashring, so the service is unreachable by the client.""

have you ever experienced this?

[evilaliv3]

@brassy-endomorph : can you upload the file /var/globaleaks/log/globaleaks.log?

[meejah]

I have seen behavior like that occasionally. I believe I've seen it on a service that "just tor config" (the txtorcon and carml onion services both run via txtorcon though).

I believe I've seen discussion about a similar issue in tor IRC as well -- would be good to have more data! :)

[evilaliv3]

Thank you @meejah !

Would you please clarify what you mean with "I believe I've seen it on a service that "just tor config"; ?

Currently in our codebase we just spawn Tor via Txtorcon and we set up some ephemeral hidden services.
Is there any logic you recommend we could periodically implement to check that the Tor process is functioning to eventually try to tear it down and re-spawn automatically in case of malfunctions?

Is the TxTorcon onion service stable or time to time do you incurr in the need for restarting your app?

[evilaliv3]

@meejah, while trying to debug i noticed one possible defect in our code that i dont know if could be cause of some issues.

We were launching our ephemeral onion service with with EphemeralOnionService.create(reactor, config, [hs_loc], private_key=key) by passing a key version 3 but i notice now that without passing a parameter version=3, the TorConfig will be set on version. Maybe this could end on some malfunctions?

[brassy-endomorph]

can you upload the file /var/globaleaks/log/globaleaks.log?

The only think that could possibly be relate to this is:

2023-12-08 03:48:33+0000 [-] [D] Fetching list of Tor exit nodes
2023-12-08 03:48:33+0000 [-] [E] Job ExitNodesRefresh died with runtime 321.0964 [low: 317.0000, high: 423.0000]
2023-12-08 03:48:33+0000 [-] Traceback (most recent call last):
2023-12-08 03:48:33+0000 [-]   File "/usr/lib/python3/dist-packages/globaleaks/jobs/job.py", line 49, in run
2023-12-08 03:48:33+0000 [-]     yield self.operation()
2023-12-08 03:48:33+0000 [-] twisted.internet.error.ConnectionRefusedError: Connection was refused by other side: 111: Connection refused.
2023-12-08 03:48:33+0000 [-] [E] Unhandled exception raised:
2023-12-08 03:48:33+0000 [-] [E] twisted.internet.error.ConnectionRefusedError Connection was refused by other side\n\nTraceback (most recent call last):\n\n  File "/usr/lib/python3/dist-packages/globaleaks/jobs/job.py", line 49, in run\n    yield self.operation()\n\ntwisted.internet.error.ConnectionRefusedError: Connection was refused by other side: 111: Connection refused.\n

This happens once per 30 minutes. The only other logs are about starting/stopping factories or sending emails.

[evilaliv3]

@brassy-endomorph: thank you, this is actually not related to the onion service. it is related to the fetching of Tor exit nodes that is then used to check if a whistleblowing is using Tor or not.

Do you feature some outgoing firewall rules in your setup or is globaleaks is free to fetch the following resource?
https://check.torproject.org/exit-addresses

[brassy-endomorph]

I can curl that URL and send TCP and UDP to the first 3 IP addresses listed in the file. The firewalls allow al outbound traffic.

[evilaliv3]

Thank you @brassy-endomorph; i will try to see if this is happening on other servers, please feel free to reach out to me on community.globaleaks.org on this matter.

[evilaliv3]

@brassy-endomorph : does it work for you if you set NETWORK_SANDBOXING=0 in /etc/default/globaleaks and you restart the application?

@rglauco just tested this and it seems to work.

If it is confirmed we should just work revising the firewall rules defined inside the init script of GlobaLeaks

[meejah]

Is the TxTorcon onion service stable or time to time do you incurr in the need for restarting your app?

It certainly does get restarted periodically (e.g. reboots) but I haven't noticed it being more or less stable than other services. Could speculate on things, but running via txtorcon is extremely similar to running from a shell so it's hard to imagine anything Python / Twisted / txtorcon specifically. I've run many Twisted things in production and they're generally pretty well-behaved and stable.

By "just Tor config" I meant that I have at least one service that isn't run via txtorcon (and it occasionally isn't working). I haven't tried to debug this so it's not clear to me if it's "a tor thing" or "networking thing" or what.

I'll keep an eye on this issue if more details come to light!

[evilaliv3]

Thank you @meejah !

@brassy-endomorph @gabrielelakhal @davidebiani would you like to give it a try replacing /etc/init.d/globaleaks with this revised script? https://raw.githubusercontent.com/globaleaks/GlobaLeaks/devel/debian/globaleaks.init

As soon that the patch is confirmed we may release issuing a bugfixed release. Thank you everyone!

[evilaliv3]

@brassy-endomorph : where are you gone? :)

[brassy-endomorph]

Replacing the init.d script with the one you provided and leaving networking sandboxing enabled has fixed the issue. Sorry for the slow reply. Day job and all.

[brassy-endomorph]

Well now I'm getting inundated with hundreds of emails:

Platform: xxx
Host:  (xxx.onion)
Version: 4.13.18

twisted.internet.error.ConnectionRefusedError Connection was refused by other side

Traceback (most recent call last):

  File "/usr/lib/python3/dist-packages/globaleaks/jobs/job.py", line 49, in run
    yield self.operation()

twisted.internet.error.ConnectionRefusedError: Connection was refused by other side: 111: Connection refused.