Protocol errors cause dropped cells

[asn-d6]

I have a public HS where I ran vanguards for testing. There is really no chance someone is gonna do a targetted attack on it because it's just a test HS.

I recently got:

WARNING[Fri Feb 15 13:48:21 2019]: Possible attack! Got a dropped cell on circ 969.
NOTICE[Fri Feb 15 13:48:21 2019]: We force-closed circuit 969
WARNING[Sat Feb 16 04:52:22 2019]: Possible attack! Got a dropped cell on circ 1292.
NOTICE[Sat Feb 16 04:52:22 2019]: We force-closed circuit 1292

What could be happening here?

[mikeperry-tor]

What Tor version was this attached to? And what vanguards commit?

[asn-d6]

NOTICE[Wed Feb 13 13:16:24 2019]: Vanguards 0.3.0 connected to Tor 0.4.0.1-alpha-dev (git-320c52e89d0d950) using stem 1.7.1`

The HS sits idle but gets visitors from people crawling the web (its address is public on the net).

Since then I've got this a few more times:

WARNING[Fri Feb 15 13:48:21 2019]: Possible attack! Got a dropped cell on circ 969.
NOTICE[Fri Feb 15 13:48:21 2019]: We force-closed circuit 969
WARNING[Sat Feb 16 04:52:22 2019]: Possible attack! Got a dropped cell on circ 1292.
NOTICE[Sat Feb 16 04:52:22 2019]: We force-closed circuit 1292
WARNING[Sun Feb 17 02:15:26 2019]: Possible attack! Got a dropped cell on circ 1806.
NOTICE[Sun Feb 17 02:15:26 2019]: We force-closed circuit 1806
WARNING[Sun Feb 17 12:46:24 2019]: Possible attack! Got a dropped cell on circ 2037.
NOTICE[Sun Feb 17 12:46:24 2019]: We force-closed circuit 2037
WARNING[Mon Feb 18 03:17:25 2019]: Possible attack! Got a dropped cell on circ 2358.
NOTICE[Mon Feb 18 03:17:25 2019]: We force-closed circuit 2358
WARNING[Mon Feb 18 11:31:25 2019]: Possible attack! Got a dropped cell on circ 2490.
NOTICE[Mon Feb 18 11:31:25 2019]: We force-closed circuit 2490
WARNING[Tue Feb 19 11:29:27 2019]: Possible attack! Got a dropped cell on circ 2989.
NOTICE[Tue Feb 19 11:29:27 2019]: We force-closed circuit 2989
WARNING[Tue Feb 19 14:05:27 2019]: Possible attack! Got a dropped cell on circ 3047.
NOTICE[Tue Feb 19 14:05:27 2019]: We force-closed circuit 3047
WARNING[Thu Feb 21 08:46:30 2019]: Possible attack! Got a dropped cell on circ 4077.
NOTICE[Thu Feb 21 08:46:30 2019]: We force-closed circuit 4077
WARNING[Thu Feb 21 14:39:30 2019]: Possible attack! Got a dropped cell on circ 4187.
NOTICE[Thu Feb 21 14:39:30 2019]: We force-closed circuit 4187
WARNING[Fri Feb 22 06:08:31 2019]: Possible attack! Got a dropped cell on circ 4611.
NOTICE[Fri Feb 22 06:08:31 2019]: We force-closed circuit 4611

I now switched to debug logging.

[mikeperry-tor]

Ok, I have collected a bunch of these. There are three classes of them so far:

1. PATH_BIAS_TESTING circuits that get an extra cell. The dropped cells I've seen so far are RELAY_COMMAND_INTRO_ESTABLISHED, and are the response from the intro point after we've switched to path bias mode and sent the probe. This arrives before the probe response. From reading the source, these can also happen with RELAY_COMMAND_RENDEZVOUS_ESTABLISHED and RELAY_COMMAND_INTRODUCE_ACK.

2. Actual INTRODUCE1 cells that are replayed by my Tor client upon retrying introductions. They look like bugs. Here are server side logs:
   Mar 07 11:10:17.000 [info] hs_circ_handle_introduce2(): We received an INTRODUCE2 cell with same REND_COOKIE field 1 seconds ago. Dropping cell.
   Mar 08 10:22:18.000 [warn] Possible replay detected! An INTRODUCE2 cell with thesame ENCRYPTED section was seen 2 seconds ago. Dropping cell.
   Mar 08 12:05:58.000 [warn] Possible replay detected! An INTRODUCE2 cell with thesame ENCRYPTED section was seen 0 seconds ago. Dropping cell.
   Mar 08 14:17:21.000 [warn] Possible replay detected! An INTRODUCE2 cell with thesame ENCRYPTED section was seen 59 seconds ago. Dropping cell.

3. ntor handshake failures during extend to the client's RP. These may be due to rotating onion keys?
   Mar 07 09:51:28.000 [info] onion_skin_ntor_client_handshake(): Invalid result from curve25519 handshake: 4
   Mar 07 09:51:28.000 [info] circuit_mark_for_close_(): Circuit 3610696346 (id: 12674) marked for close at src/core/or/relay.c:1731 (orig reason: 1, new reason: 0)
   Mar 07 09:51:28.000 [info] circuit_build_failed(): Couldn't connect to the client's chosen rend point "" (last hop failed).
   Mar 07 09:51:28.000 [info] can_relaunch_service_rendezvous_point(): Attempt to build a rendezvous circuit to [scrubbed] has failed with 1 attempts and expiry time 1551952317. Giving up building.

[mikeperry-tor]

Ok I have filed https://trac.torproject.org/projects/tor/ticket/29786 for 1, https://trac.torproject.org/projects/tor/ticket/29699 for 2, and https://trac.torproject.org/projects/tor/ticket/29700 for 3.

There may be more cases of buggy "normal" client behavior than the ones I have found.

In the meantime, I think the best we can do is demote this log to notice, and maybe give select path bias circuits a free dropped cell (so we don't close them early and influence path bias counts).

[mikeperry-tor]

Ok asn, others: That commit has workarounds for the found cases so far. Can you please pull master and look for more WARNs? Those will be new instances.

[asn-d6]

OK done. Will let you know of results.

[mikeperry-tor]

Ok there are a 4th class of these: mystery client-side ones that do not cause Tor to emit any log lines, even at debug level. I have filed https://trac.torproject.org/projects/tor/ticket/29927 for them. Committing workarounds shortly.

[asn-d6]

Just FTR with vanguards ad3f5163ce629fbef04ea37d2d5203649eb441e5 I still got a:

NOTICE[Fri Mar 22 06:35:38 2019]: Tor bug #29699: Got a dropped cell on circ 962. (in states HS_SERVICE_INTRO HSSI_ESTABLISHED HS_SERVICE_INTRO HSSI_CONNECTING)
NOTICE[Fri Mar 22 06:35:38 2019]: We force-closed circuit 962

[mikeperry-tor]

Just FTR with vanguards ad3f5163ce629fbef04ea37d2d5203649eb441e5 I still got a:

NOTICE[Fri Mar 22 06:35:38 2019]: Tor bug #29699: Got a dropped cell on circ 962. (in states HS_SERVICE_INTRO HSSI_ESTABLISHED HS_SERVICE_INTRO HSSI_CONNECTING)
NOTICE[Fri Mar 22 06:35:38 2019]: We force-closed circuit 962

Yes, this is intentional. In this case, I chose to close the circuit since these happen infrequently, but I did not want to allow an adversary to send a lot of them down the same path. Because of both of these, I put the log message at NOTICE level, which we define in our README as being the level for "False positives that we know about."

If you think this is still too scary, I'm open to suggestions.

[mikeperry-tor]

Got another instance of case 4. It turns out any circuit can get a dropped cell during construction, and then fail with END_CIRC_REASON_TORPRORTOCOL. There is also no log message for this. Possible ntor handshake failure, similar to the rend case? Not sure tho. Workaround coming

[mikeperry-tor]

Fixed in vanguards 0.3.1 that was released today.