Skip to content

LGC-2026-08-13T084949

LGC-2026-08-13T084949: PIR: Forward extras on extracted profiles (#9353)
@noisysocks noisysocks tagged this 13 Aug 00:07 
Task/Issue URL:
https://app.asana.com/1/137249556945/task/1216773212741753
Tech Design URL (if applicable):

https://app.asana.com/1/137249556945/project/481882893211075/task/1216773212741757
API Proposals URL(s) (if applicable):

### Description

Support `extras` in extracted profiles and addresses, pass them along to
`fillForm`, and update stored profiles with the freshly scraped profile.
This lessens the need for an app release when an opt out form requires a
new field we don't have.

### Steps to test this PR

1. Point C-S-S to [Support extracting extras from profiles and addresses
content-scope-scripts#2907](https://github.com/duckduckgo/content-scope-scripts/pull/2907):

```
npm install @duckduckgo/content-scope-scripts@github:duckduckgo/content-scope-scripts#pr-releases/randerson/pir-extras-open-field
```

2. Build and run.
4. Open Settings → Pir Dev Settings → Broker Config.
5. Select `cyberbackgroundchecks.com` and replace the JSON with:
https://dub.duckduckgo.com/duckduckgo/dbp-api/blob/2fb2bb29f5d3f67bc6c70098ca36db7ca4a100b9/dbp-json/data/json/cyberbackgroundchecks.com.json
6. Complete a scan and opt out: Debug Scan → Run opt out → PIR Email →
Debug opt out.
7. The street and zip fields in the post-confirmation opt out form
should be filled from data that was extracted during the scan.


<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Medium Risk**
> Touches PIR local storage, scan refresh semantics, and opt-out
identity matching; changes are well-tested but incorrect merge or key
logic could affect pending opt-outs or removal detection.
> 
> **Overview**
> Adds **`extras`** on extracted profiles and addresses so broker
configs can scrape arbitrary fields without a native release, and passes
the fuller profile through to C-S-S for **`fillForm`**.
> 
> On repeat scans, **`saveExtractedProfiles`** refreshes rows that
already match the DB unique key (profile query, broker, name, URL,
identifier) instead of ignoring them, merging extras so keys missing
from a new scrape keep stored values. Profile identity for
removals/reappearances uses a narrower key and **`matches`** ignores
address extras so new scraped fields don’t look like a different person.
> 
> **`ExtractProfileSelectors`** becomes a passthrough map so extract
profile config isn’t dropped at parse time. Room bumps to **v17** with
an **`extras`** column, Moshi map converters, and migration tests.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
71d26d75b1cfb1de6e6c01f9ed8005f44fc8a619. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Assets 2
Loading