IPv6 Support

[bretg]

IPv6 is becoming more common, and Prebid Server needs some updates to handle it in a privacy-appropriate way. Note that IPv6 addressing is quite different than IPv4 - there are a number of good resources on the net to understand them.

Config

We should support the following host configuration options:

• A setting to anonymize the network piece when we need to protect user location. Suggested config: ipv6.anon-left-mask-bits. Default to 56 bits. Throw an error if this isn't >=56 and <=32.
• A setting to remove a number of bits on the right regardless of whether it's a situation that calls for explicit privacy. The idea is that no one downstream needs the whole device MAC address. Suggested config: ipv6.always-mask-right. Default to default 64 bits
• A setting to define the network ranges considered "private" and therefore ignored. Suggested config: ipv6.private-networks. If not specified, default to [::1/128, fc00::/7, fe80::/10]

Proposed Algorithm

1. If the device.ip field is present in the incoming request, validate it.
   1. If not valid or is determined to be a private network, remove.
   2. If the current privacy environment requires, anonymize the address. (see algorithm below)
2. If the device.ipv6 field is present in the incoming request, validate it.
   1. If not valid or is determined to be a private network, remove.
   2. If the current privacy environment requires, anonymize the address. (see algorithm below)
   3. Even if we don't need to anonymize the address, check the always-mask-right configuration to see if we need to remove any bits off the right side.
3. If device.ip and/or device.ipv6 are validated, we're done. We're going to trust the upstream clients to provide these correctly.
4. If we don't have either, then we need to find one. Start with the True-Client-IP header. If it exists:
   1. Validate, anonymize, and mask as needed
   2. Write to the appropriate device.{ip,ipv6} field
5. Else, continue by parsing the comma-separated IP values in XFF and looping through the addresses from left to right
   1. If the address is IPv6, check to see if matches one of these the configured IPv6 private networks
      1. If so, skip it and go to the next address
      2. If the current privacy environment requires, anonymize the address. (see algorithm below)
      3. Even if we don't need to anonymize the address, check the always-mask-right configuration to see if we need to remove any bits off the right side.
      4. If the OpenRTB device.ipv6 field isn't already present, write the new address. If there was an address already, and there's a post-mask difference between the addresses, log a warning.
      5. Break out of loop
   2. Else if the address if IPv4, check to see if it's an IPv4 private network
      1. If so, skip it and go to the next address
      2. If not, then check the current privacy environment. If we need to anonymize the IPv4 address, then do so.
      3. If the OpenRTB device.ip field isn't already present, write the new address. If there was an address already, and there's a post-mask difference between the addresses, log a warning.
      4. Break out of loop
6. Else if no address was found in XFF, look in the following locations in this order: X-Real-IP header, and HTTP client socket. If we find a valid public address:
   1. Anonymize and mask as needed
   2. Write to the appropriate device.{ip,ipv6} field
7. If there's still nothing, neither device.ip nor device.ipv6 will be set. Log a warning.

Anonymize IPv6:

1. Check the anon-left-mask-bits config and mask the appropriate number of bits off of the first 64 bits
2. Replace the last 64 bits with 0's.

Anonymize IPv4:

1. set the last octet to zero

Examples

1. If we need to privacy mask address 1111:2222:3333:4444:5555:6666:7777:8888
   the default result would be a 56 bit netmask -- 1111:2222:3333:44::
   if anon-left-mask-bits was 48, the result would be 1111:2222:3333::

2. Even if we didn't need to mask address 1111:2222:3333:4444:5555:6666:7777:8888, if always-mask-right is 64, the result is 1111:2222:3333:4444::

Since there multiple ways to validate and manipulate IPv6 addresses, we ought to research validation libraries for Go and Java.

[SyntaxNode]

This is a great start. I have some follow-up questions:

1. If the request has a valid non-private ip, why should we continue to parse the headers, per section 4-*-d?

2. If the request has a valid non-private ip or ipv6, but not both.. should we still parse the headers to try and fill in the missing one? I'm actually not even sure this is a valid use case. Is there some exotic scenario when a request would have both? Maybe the javascript fills in an ipv4 and then the actual call is made over ipv6 or some such?

3. The OpenRTB 2.5 spec calls to anonymize an ipv6 address by removing the last 32 bits. We do this today for COPPA and 'downgrade' to 16 bits for GDPR/CCPA/LMT. Why the proposal to change this to 56 bits? .. and why should this be host configurable?

4. How would bidders feel about PBS always rounding off the ipv6 address when there's no privacy-related reason to do so (request is not in scope of COPPA/GDPR/CCPA/LMT?

[bretg]

If the request has a valid non-private ip or ipv6

The question comes down to whether we should trust what's passed in. Let's discuss in committee. Spoofing the value in the request is easier than spoofing the XFF. Unclear there's a "right" answer here.

The OpenRTB 2.5 spec calls to anonymize an ipv6 address by removing the last 32 bits

The Rubicon security team has stronger guidelines. First, the rightmost 64 bits is all MAC address and should be completely dropped all the time in our environment. Second, the leftmost 64-bits is the network -- we want to mask off the last 8 bits there. (64-56=8)

How would bidders feel about PBS always rounding off the ipv6 address when there's no privacy-related reason to do so

The Rubicon exchange does this today, which is why we want our PBS to do the same. The idea is that the MAC address isn't useful for targeting. This setting is proposed as a configuration so other host companies can be more lenient.

[SyntaxNode]

The question comes down to whether we should trust what's passed in. Let's discuss in committee. Spoofing the value in the request is easier than spoofing the XFF. Unclear there's a "right" answer here.

Sounds good to me.

The Rubicon security team has stronger guidelines. First, the rightmost 64 bits is all MAC address and should be completely dropped all the time in our environment. Second, the leftmost 64-bits is the network -- we want to mask off the last 8 bits there. (64-56=8)

Gotcha. Is there a use case for removing bits from the network and leaving the mac address component in place? I'm wondering if a single network bit mask option would keep configs a bit more straightforward. If we really want to provide the option though and agree it's of potential use, I'm personally fine with it. We can discuss this in the committee as well.

.The Rubicon exchange does this today, which is why we want our PBS to do the same. The idea is that the MAC address isn't useful for targeting. This setting is proposed as a configuration so other host companies can be more lenient.

Cool. Looks like Google and others do this as well. I'll get guidance from Xandr for the committee discussion.

Thinking a bit more about OpenRTB request validation. I'm not worried about an ipv6 field really having an ipv4, but I wonder if we should support an ipv4 having an ipv6 and gracefully promote it to the ipv6 appropriate field.

[bretg]

We discussed in committee. Updates made above.

[bretg]

Apparently 2001:db8::/32 is seen in the wild -- some host companies may want to consider it a private network.

[bretg]

This has been released in PBS-Java.