Add Huawei to blacklist

[Lee-Carre]

I've observed several instances of such SSIDs while capturing data with various apps.

In my experience, they're always mobile phones.

[n76]

@Lee-Carre I no longer have an Android device to test builds much less use this software on a daily basis to assure it is working correctly. It seems from your comments that you are an active user of it. Would you like to take over this project?

[gdt]

I am inclined to try to help, but I do not and won't have anything to do with the play store. From my viewpoint all that matters is f-droid and caylxos integration :-)

[n76]

I am inclined to try to help, but I do not and won't have anything to do with the play store. From my viewpoint all that matters is f-droid and caylxos integration :-)

Sounds like a kindred spirit. I have spent and continue to spend considerable effort to avoid Google and the only "store" I made any of my Android available through was F-Droid.

If you wish to take over this project and/or the wifi backend, I have no objections.

[Lee-Carre]

It seems from your comments that you are an active user of it. Would you like to take over this project?

Firstly, thankyou for the offer. I'm flattered.

However, as much as I appreciate the opportunity, I have major reservations about taking on that kind of responsibility

• I've very little experience with full-on development (more of a clueful user, with experience in systems & networking, who's dabbled in a little scripting, at this point) or managing a tech project (e.g. I'm a git newbie (or is that newbie git? 😋)). I've a lot of learning yet to do.
• my personal circumstances are messy & complex, due to factors beyond my control (and not likely to change in the foreseeable future). I would be very limited in what I could practically do, and can't offer any likelihood of reliability (e.g. responding in a timely fashion)

Having crossed paths with @gdt several times on related projects, he seems a far better candidate to inherit the project & its maintenance. I definitely approve of the anti-Google (more specifically anti-proprietary) & pro-libre sentiments.

If & when my reservations become non-issues, I'll gladly contribute & assist @gdt (or whoever he, or you, nominate), in future.

[Helium314]

Semi-related: I've been working a bit on DejaVu in the past weeks, because I noticed things are getting slow if the database is large (50 MB in my case)
You might want to have a look at https://github.com/Helium314/DejaVu

What has changed

• fixed a few bugs, e.g. blacklist not being used because RfEmitter.setNote is never called
• changed database format to reduce size and accelerate queries
• some performance improvements when getting expected emitters
• (partially) migrate to kotlin, simply because I'm much more familiar with kotlin than with java
• prepare capability to use more emitter types: bluetooth, 6 GHz WiFi, 5G; split mobile into GSM / CDMA / LTE /... (but not acually in use)
• some more small changes / tuning attempts
• probably introduced some not yet discovered bugs

[Lee-Carre]

@Helium314

I've been working a bit on DejaVu in the past weeks

What has changed

Sounds promising 👍. Looking forward to seeing those released. I started watching your fork; thanks for citing it.

things are getting slow if the database is large

While I can't comment on code being (in)efficient (though I'm prompted to think of determining the Big-O characteristic of algorithms), I've a little experience with DBs.

• Presuming said database is SQLite, a simple & easy win should be to use indexing especially of the frequently-used criteria (e.g. for SELECTs doing lookups).
• Ensuring that SELECTs are well-crafted (small changes can sometimes make significant reductions to execution-time, also beware SELECT *), should also help.
• Having the backend perform the occasional VACUUM on the DB would be sensible. Perhaps after every N new / updated entries (instead of time-based).
• [Edited to add:]
  • the lesson of Bobby Tables
  • How Fucked is my Database?

[Helium314]

Thanks for the hints!
Index on latitude / longitude / emitter type is already in use (for querying nearby emitters), and the primary key for the other queries.

I thought about switching to bounding boxes (storing north, south, east, west) instead of center and N-S / E-W radius. This would allow to properly query all emitters whose bounding boxes insersects with our location, but then queries are considerably slower (even with index)

[Lee-Carre]

Thanks for the hints!

Welcome. Added a couple more, since.

I thought about switching to bounding boxes (storing north, south, east, west) instead of center and N-S / E-W radius. This would allow to properly query all emitters whose bounding boxes insersects with our location, but then queries are considerably slower (even with index)

Even if it were faster:

• does it align with the goal of the backend (the design spec, if there were one)?
• (estimated) centre & radius are exactly what the backend is concerned about
• with centre & radius, one can be updated without affecting the other, whereas with a bounding box, you've always got to update the quad-set (can't think of the proper term for the 4 nodes at the corners)
• emitters are very likely to (appear) to have omni-directional antennas (at least in the horizontal plane, which is (mostly?) all we care about), and thus (ignoring signal-attenuating material in the signal path) are likely to have circular-shaped coverage (rather than rectangular) — in the case of telephony cells; while the emitters themselves aren't (usually) omni-directional, what (I presume, off the top of my head) we more likely care about is the centre of the cell (covered by multiple cell sites)
• 4 pairs of coördinates is very likely to be more bytes to store, for each emitter, than centre (pair of coördinates) & 2 radii (an integer each) — the difference becomes apparent when multiplied by tens of thousands of emitters
• if a bounding box were needed (assuming that doing geometry computations with circles wasn't viable), then compute it from the centre & radius (only for the set of emitters you currently care about), but continue to store centre & radius. If the bbox quad is needed for other near-term calculations, then cache the most recent N in memory.
• when deciding what to optimise for, it's likely to be lookups, both for checking currency / recentness of stored data while collecting, and for yielding positions via UnifiedNLP. Adding emitters (since that'll be less often) can afford to be slower (dump them into a queue, then it matters even less).
• more generally, I'm reminded of the many wise lessons in The Art of Unix Programming, such as about avoiding complex techniques if simpler ones will do. Complexity is the enemy of robustness & transparency. If brute-force is sufficiently performant (considering use-case, dataset (size), and hardware), then use that. Optimise only where needed. Excellent explanations of all this, and more, in the book (which is available, gratis, as HTML, on the site).

[gdt]

This discussion is now off topic for the PR (I saw with zero authority), which points out that the project really needs a mailinglist, the lack of which is part of github disease. But, as someone who has handled maintenance for projects, I think any restart would need to focus on bringing up to modern API revisions, fixing real bugs, and fixing things that are widely causing real pain. And, I have a very strong leaning to "simpler is better".

[Lee-Carre]

discussion is now off topic for the PR (I sa[y] with zero authority)

😄 I was mindful of off-topic-ness, too.

Obligatory citation of Posting And You….

the project really needs a mailinglist, the lack of which is part of github disease.

Mailing lists aren't easily accessible to all. I'd suggest either enabling the Discussions section feature for the repo, or a meta-issue.

[Helium314]

Still off-topic, but...

I don't see how storing bounding boxes would be much more complicated in any way than storing center and radii.
In both cases we have 4 reals, which SQLite stores using 8 bytes to my knowledge, so no difference in size (radii could be stored as integers, but aren't).
[Edit: I did not consider that index on 2 more columns probably increases size]

The bounding box is created from center and radii when the emitter is loaded from database, so there is no difference once the emitter is loaded from database. And in the database it would be good to use whatever is more useful for our queries.

The only queries for the emitter location are for getting emitters we expect to see.
I think checking whether our position is inside the bounding box of an emitter is a good way to check whether we should see it. In my opinion it's better and actually more simple than getting the typical range of an emitter type, and query all emitters of that type in a radius of typical range around us (repeat for each type).
I think so, because the typical range query finds "expected" emitters that we should not find because their range is less than typical range (according to our data). And it doesn't find emitters with more than typical range which we actually would expect.