What additional request attributes should we capture?

[johnspurlock-skymethod]

Right now, OP3 follows the principle of Data minimization, and only captures/stores the small subset of the request attributes that it needs to serve back in the API, no more.

We have many other request-level attributes available to us, but now are completely ignoring/dropping them. Some of these attributes should probably be considered safe to process as they have little to no listener privacy impact, and would make our eventual show/episode-level analytics much more useful.

I'd like to arrive at a consensus as to which of the following attributes are "no-brainers" to start storing/returning, those that are "on-the-line", could go either way, and those we should always avoid storing. The goal is to provide maximum utility without going over the creepy line wrt listeners' expectations.

Here are the attributes we are not currently capturing for each redirect request, but could in the future:

• asn: (number) the ASN of the incoming request (e.g. 395747)
• asOrganization: (string) The organization which owns the ASN of the incoming request (e.g. Google Cloud)
• country: (string) two-letter country code (e.g. US)
• isEUCountry: (boolean) Whether the country is in the EU
• city: (string) (e.g. Austin)
• continent: (string) (e.g. NA for North America)
• latitude: (string) (e.g. 30.27130)
• longitude: (string) (e.g. -97.74260)
• postalCode: (string) (e.g. 78701)
• metroCode: (string) Metro code (DMA) (e.g. 635)
• region: (string) If known, the ISO 3166-2 name for the first level region (e.g. Texas)
• regionCode: (string) If known, the ISO 3166-2 code for the first-level region (e.g. TX)
• timezone: (string) (e.g. America/Chicago)

To me: country, continent, region/regionCode, and timezone seem like no-brainers.

I think metroCode would also be broad enough to be non-identifying but still very useful for folks building on top of OP3.

asn would be potentially useful for tagging traffic coming from cloud services in the future.

What does everyone think?

[jsonperl]

timezone could be nice for backing out questions like "what time of day do folks listen to my episode" without going through region gymnastics.

[johnspurlock-skymethod]

ah that's true, we could even derive a localRequestTime in the data API using the associated timezone. Hmm

[jamescridland]

Yes, I see that being really useful.

(Technically: "what time do people download my episode", which isn't the time they listen to it.)

[evoterra]

I’m not sure how accurate lon/lat and other values are anyhow. Cell traffic, corporate routers… so I’d be shy of data that’s not a certainty.

[johnspurlock-skymethod]

Thanks Evo, makes sense to keep the list of captured attributes small and stable. Garbage in, garbage out

[tomrossi7]

I don't have any hesitations about privacy with that dataset. Some of it may be redundant (e.g. a country will only ever be on one continent), but maybe you just want to make it nicer for the API to do some of those lookups for them?

I’m not sure how accurate lon/lat and other values are anyhow.
Geolookup varies greatly on how accurate it is. Sometimes is very granular and accurate and other times its only at the region level or very granular, but a totally different city 🤣. At the end of the day, all location data should be taken with a grain of salt.

[johnspurlock-skymethod]

Thanks - I see nothing but downsides to keeping lat/lons at this point

[jamescridland]

Country, continent, timezone, and possibly regioncode for me (I wonder whether that's just a US thing). I'd be very wary of using anything more granular: while we understand that lat/lon or name of place is only approximate, the optics of it are that "OMG we are being tracked", and I'm not sure that helps anyone.

For me, for example, I'd expect "Brisbane" would show up, but in reality, knowing the datasets, it'll be the small suburb I live in, Bardon, which has a very low population of just 9,500 people. (Let's not forget - the podcast hosting company can absolutely show that to their customers if they want, but I'm a little more wary of this being open data available everywhere. The fewer objections people have to joining OP3, the better, as far as I'm concerned.)

asn may be useful to help combat fraud, and I think it worth including that. That could directly lead back to the (erroneous, in my view) "disregard everything from Amazon AWS" rule that the IAB has.

Other suggestions: app and os would be great, though that would require matching work done with the OPAWG list or similar, and would, I think, be out of scope.

[jamescridland]

And from my fixed connection/Chrome (this is it in full, there's nothing really to censor)

{"clientTcpRtt":13,"longitude":"153.00420","latitude":"-27.47500","tlsCipher":"AEAD-AES128-GCM-SHA256","continent":"OC","asn":4764,"clientAcceptEncoding":"gzip, deflate, br","country":"AU","tlsClientAuth":{"certIssuerDNLegacy":"","certIssuerSKI":"","certSubjectDNRFC2253":"","certSubjectDNLegacy":"","certFingerprintSHA256":"","certNotBefore":"","certSKI":"","certSerial":"","certIssuerDN":"","certVerified":"NONE","certNotAfter":"","certSubjectDN":"","certPresented":"0","certRevoked":"0","certIssuerSerial":"","certIssuerDNRFC2253":"","certFingerprintSHA1":""},"tlsExportedAuthenticator":{"clientFinished":"ef184251b452ca0d35085abeb349dc0f443199af1fac2e6ca7b45c613e45602a","clientHandshake":"0eb8af57b4619919ee97b7b2fc3d48ab168ebb0f9fc0f6f3b02bf138923d31ea","serverHandshake":"a0731ab0ddc37bcc81ce96625614bffc77e40e908a975c2deb66d18c532cc583","serverFinished":"d6121a657abd6dcfa780b9487f421c42e0b0a944ba959f95fb23dccfe32b9e5d"},"tlsVersion":"TLSv1.3","colo":"BNE","timezone":"Australia/Brisbane","city":"Brisbane","httpProtocol":"HTTP/2","edgeRequestKeepAliveStatus":1,"requestPriority":"weight=256;exclusive=1","botManagement":{"corporateProxy":false,"staticResource":false,"verifiedBot":false,"score":91},"clientTrustScore":91,"region":"Queensland","regionCode":"QLD","asOrganization":"Aussie Broadband","postalCode":"4064"}

The postalCode is not correct (4064 is a neighbouring one), but that's a population of 8,500 and I would, again, suggest that this is too small to include.

The lat/lon again places me in the middle of the Brisbane River, but in Milton.

[jamescridland]

Interestingly, a "where is my IP" lookup places my IP (IPv6) in Melbourne VIC, Australia - 1,021 miles away and an 18hr 27min drive from here.

[johnspurlock-skymethod]

What do you think about city, then? Too creepy? I think US metro seems useful and broad enough, but it's US only.

City, of course, would be useful for building "hot in cleveland" type services - where a massive state like Queensland might not be as germane

[jsonperl]

Living in Cleveland, I would welcome this functionality! 🤣

[johnspurlock-skymethod]

Right on (I grew up there): too bad github does not have a "river on fire" reaction emoji

[drpancake]

I agree that storing latitude and longitude creates more of a liability (in terms of perception of the project) than it's worth. City-level granularity seems fine. The other attributes look useful and non-controversial to me.

[daveajones]

I would be careful about including asn as that can help an attacker derive the secret key used to hash the IP's. I think you are using SHA-1 for the hash, so any extra bit of input info will make cracking it easier.

[daveajones]

An alternative would be to just mark the requests as being from a hyperscaler or not. Or maybe mark the source as residential/hyperscaler/business/etc. I know some of that info is available as lists, but not sure how fine grained it is.

[johnspurlock-skymethod]

Interesting, right it's a hmac-sha1, so it would still be very difficult, but more info signal to be sure. asn could be a field that we capture (and disclose), but use only for tagging, and not returning directly in any api calls.

[johnspurlock-skymethod]

Alright, just pushed a release yesterday that captures more of these non-listener-identifying attributes on every request served via OP3.

https://op3.dev/releases#2022-10-20

Thanks to everyone for chiming in!