Implement IMSI-Exposed analyzer

[untitaker]

For discussion see #398

I don't think we can do anything useful with the exposure ratio idea,
since we are just one device and have 1-2 connections at the same time.

Implementing some kind of analyzer based on aggregate data from many
Rayhunter devices would be a fun project for the Meshtastic crowd.

This analyzer may not be useful. I mainly implemented it to see how much
noise this dumbed down version produces.

[untitaker]

if we think we want the full analyzer for rayhunter-check, note that MAC-LTE parsing is not implemented and it seems like quite some work to get it going. we need it for this line in marlin:

lte_rar_filter = '(mac-lte.rnti-type == 2)'

[MatejKovacic]

I have got an idea. What if we draw a timeline chart of IMSI requests? For instance bar chart with 5/10/whatever minute interval? Then we will be able to see a timeline where there is a "burst" of these requests.

Would that be helpful?

[untitaker]

@MatejKovacic what would the threshold be to say "there's an imsi catcher"? just one of those requests should be enough to be exposed. it's only when you observe many different devices' traffic that you can observe many imsi requests from the imsi catcher and make some kind of statistic out of it.

or did i misunderstand the paper?

[oopsbagel]

I think for analyzer changes we should make sure we get Cooper or Will to sign off.

[alliraine]

Code looks good. I wonder if we should hide this option and "experimental" tag or section.

[untitaker]

it's off by default -- if users intentionally turn it on and then report bugs that it's too noisy, i think that's the outcome we'd want. i've been running aroudn with it for a bit and got zero hits so far here. but if it's different in other countries it's good to know.

[cooperq]

I have serious concerns about this one. Our whole goal is to have as few false positives as possible and I think that this heuristic is almost designed to generate false positives.

[untitaker]

Our whole goal is to have as few false positives as possible

Rayhunter already has a very high false-positive rate at least in my area, and now we have a report of an attack which Rayhunter wouldn't have been able to catch. If we want to make progress on either of those issues, I believe there needs to be room for experiments that users can opt into, as we cannot test all kinds of locations pre-release. I'm open to suggestions on how the UX for those experiments should look like, whether they should not just be disabled by default but also hidden in the UI, etc. Anyway like I said I've been running around with this in Austria for a day and it hasn't caused issues, it would be good to know if that is an outlier experience.

I think that this heuristic is almost designed to generate false positives.

I agree that this is almost certainly too dumb, but unless we see pcaps of false positives there is no data-driven way to be able to refine this. I wish we were in a position like the Marlin folks where we can implement heuristics based on traffic information from many devices within the area.

[MatejKovacic]

I guess, we can have several analysers and people can turn them on or off in the config. And some could be off by default. They could even be marked as experimental.

That would solve the problem, I believe.

[untitaker]

I guess, we can have several analysers and people can turn them on or off in the config. And some could be off by default. They could even be marked as experimental.

this analyzer is off by default, or are you suggesting to split it up as well?

[MatejKovacic]

No, I was just replying to @cooperq . I understand his concerns, but if this could be turned on/off and is off by default, I see no reason to worry.

And actually, you can have different versions of similar analysers (I would just mark them experimental) to test which version performs better.

[cooperq]

Rayhunter already has a very high false-positive rate at least in my area,

With the 2g heuristic correct? Which is why we now have the ability to control which heuristics you run, because we want to decrease false positives.

and now we have a report of an attack which Rayhunter wouldn't have been able to catch.

I've looked at the data from this report and I'm pretty convinced that the detections from marlin were a false positive.

If we want to make progress on either of those issues, I believe there needs to be room for experiments that users can opt into

I'm not opposed to having experimental heuristics that users can opt into but I don't think that the foundation of this heuristic is very good. The fact is that messages that "may expose the IMSI" such as attach reject and tracking area update messages happen super often, and most of them are benign. We could log every time the IMSI is sent by the device but again this is something that happens often and often for benign reasons. I would rather focus on heuristics that are high signal or purely informational like what tower are you connected to.

[wgreenberg]

my thinking is that this heuristic is only useful in aggregate (as in the Marlin paper), and so it's fundamentally not meaningful at runtime for a rayhunter user. as was mentioned earlier in the thread, if we somehow pooled data in real time with neighboring devices, it could start to approach what Marlin does, but that's not in scope for rayhunter.

one could theoretically run this kind of check offline by collating lots of PCAPs from different devices at the same event, but that's not even something our analyzer framework is capable of doing. so i'd agree that even if it's part of a useful SDR-based detection scheme, it's not a useful rayhunter check

[cooperq]

If we want to have more informational analyzers I'm not opposed to that but I think we should keep them as informational and not warnings.

[untitaker]

the purpose of having such an analyzer in rayhunter would be to get feedback on its false-positive rate. like i said, it does not trigger at all here. doing the same kind of analysis across many peers means preemptively collecting pcaps from all peers before anything happens, which is very difficult because of PII.

What is the difference between lowering the level to informational and having the analyzer disabled by default? from what i can tell informational effectively hides analyzer output from the UI.

you already have a feature request open for making analyzer levels configurable via the UI as well, because they are not universally appropriate. maybe this needs to be solved first before this analyzer can be useful, but it's a lot of extra work.

[cooperq]

I've given this some more thought and I'm okay with this heuristic being on by default if we do the following:

1. Rename it, these aren't IMSI exposing messages but messages detach requests or attach rejects that could potentially cause a UE to expose its IMSI. I think we should name it 'marlin heuristics' or something like that.
2. Move them from warning to informational: I don't think these are super useful on their own but they can provide context along side other warnings. I think they are important context and are useful as that for sure.
3. we already notify about EMMMessage::EMMIdentityRequest(_) packets so this is a duplicate of an existing heuristic and we should remove it. I'm open to keeping it however for the sake of implementing as much of the marlin heuristic as possible.

[untitaker]

cooper, we are talking past each other. I do not want this to be on by default. In fact I would strongly prefer it if it can stay at warning level but instead be off by default, because informational level is not very visible in the UI.

[cooperq]

yes but the point I'm trying to make is that these heuristics are just not very useful on their own. It's going to lead to a lot of false positives that I then have to spend my time digging through or worse yet some journalist to write a story about an IMSI catcher that wasn't actually there. What these heuristics are useful for is context along with other warnings, that's why I think they make better informational messages than warnings. I still want a name change for the entire heuristic but with that I think they are useful context. But I'm very hesitant to give them the level of actual warnings

[cooperq]

yes but the point I'm trying to make is that these heuristics are just not very useful on their own. It's going to lead to a lot of false positives that I then have to spend my time digging through or worse yet some journalist to write a story about an IMSI catcher that wasn't actually there. What these heuristics are useful for is context along with other warnings, that's why I think they make better informational messages than warnings. I still want a name change for the entire heuristic but with that I think they are useful context. But I'm very hesitant to give them the level of actual warnings because they are not suspicious on their own.

[untitaker]

I'll implement these changes but I really don't know how we'd mislead anybody here by having a heuristic that is off by default and very clearly marked as experimental and with "do not send reports to Signal", regardless of what the event level is. I don't think people in EU have been misled by e.g the 2G downgrade warnings to the extent you're concerned about.

I still want a name change for the entire heuristic but with that I think they are useful context.

I don't think it's fair to call this marlin heuristic since this is (theoretically) a lot more noisy

[cooperq]

You could call it the detach messages heuristic if you don't want to call it the marlin heuristic, since that's mostly what it is. (except for the identity request which we could roll into the identity request heuristic also as an informational log)

[untitaker]

I don't think it's worth upstreaming this anymore. I have a fork now for my own purposes.