Start exporting data in opencellid format

[hannosch]

I spoke with @msemm from the opencellid project last week about data sharing.

We agreed on trying to use the same data format for exports for both our projects, to make it easier for third parties to use data from either project. OpenCellId is currently changing their export format to account for network types. The documentation on the new data format should be available at http://wiki.opencellid.org/wiki/Menu_map_view#database in the next weeks.

We still maintain our view on privacy and won't share the individual observations, but only share the aggregate data set. We will export our data under public domain (CC-0) terms.

We agreed to publish data as daily snapshots and hourly differential updates at predictable url paths. In our case this will likely be a Amazon S3 url.

In order to generate hourly diffs we'll need to introduce an update time into the cell/wifi tables.

[msemm]

Hallo Hanno,

OpenCellID Version 3.3.9.1 ist seit gestern auf dem Prod.-System freigeschaltet.
Sie unterstützt jetzt alle bekannten Netzwerktypen.
Die Doku auf wiki.opencellid.org ist aktualisiert.

In Deiner Mail fehlen ziemlich viele Punkte aus meiner Zusammenfassung.
Was wird damit? Sind die gestorben?

Viele Grüße
Markus

Von: Hanno Schlichting [mailto:notifications@github.com]
Gesendet: Dienstag, 29. Juli 2014 20:27
An: mozilla/ichnaea
Cc: Markus Semm
Betreff: [ichnaea] Start exporting data in opencellid format (#282)

I spoke with @msemmhttps://github.com/msemm from the opencellid project last week about data sharing.

We agreed on trying to use the same data format for exports for both our projects, to make it easier for third parties to use data from either project. OpenCellId is currently changing their export format to account for network types. The documentation on the new data format should be available at http://wiki.opencellid.org/wiki/Menu_map_view#database in the next weeks.

We still maintain our view on privacy and won't share the individual observations, but only share the aggregate data set. We will export our data under public domain (CC-0) terms.

We agreed to publish data as daily snapshots and hourly differential updates at predictable url paths. In our case this will likely be a Amazon S3 url.

In order to generate hourly diffs we'll need to introduce an update time into the cell/wifi tables.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/282.

[hannosch]

@msemm I'm writing the private reply to the other issues right now. This is just one of the concrete work items for our service side code.

[dper]

Is there any export functionality at the moment, in any format?

[hannosch]

@dper there is no export functionality yet. This is finally the time for us to add export, and reusing the opencellid format made sense to us.

[Gitschubser]

I would not use the actual opencellid format.

Reasons:

• Opencellid collects different cellid formats. They collect the cellid (UMTS) in two different formats as one value, as LCID (with RNC) or short cellid.
  There are many duplicates in the database. MLS collects only the LCID format (RNC+cellid)?
  There should be used the LCID or CID and RNC or both.
• The value latitude and longitude from MLS is the calculated value (middle) from all measurements.
  opencellid stores the cellid as a mix with the exact position and calculated value (value changeable opencellid).
  If you use the mix from this cellids for triangulation the calculated position is false.
• The value averageSignal make in my opinion no sense.
• I miss the value range in opencellid. This is a must-have and MLS calculates this value.

Example maybe must-have for GSM/UMTS/LTE:
mcc, mnc, lac/tac, bsc/rnc, cellid, lcid, radio, latitude, longitude, range, samples, firstseen, lastseen

1: mcc = Mobile Country Code
2: mnc = Mobile Network Code
3: lac/tac = Location Area Code/Tracking Area Code
4: bsc/rnc = Base Station Controller (GSM)/Radio Network Controller (UMTS)
5: cellid = Cellid (short)
6: lcid = Cellid (long)
7: radio = radio type GSM/UMTS/LTE/CDMA
8: latitude
9: longitude
10: range = Accuracy als circle in meters
11: samples = How many samples exists for this cells, how good is the calculation
12: firstseen = timestamp first measurement
13: lastseen = timestamp last measurement

The value samples could be a indicator for the quality or any another idea?

[cpeterso]

@hannosch, we should probably ask the dev-geolocation mailing list for feedback your proposed export format before we start publishing cell data. We want the data to be useful to people. :)

[Gitschubser]

The exported format should include all required informations that are important for a offline geolocation with mozstumbler. Offline geolocation would be a good feature for mozstumbler.

[hannosch]

A couple of follow-up items are tracked in #295.