Develop (FtM 4.0)

[pudo]

Collecting the various PRs here.

Draft changelog

• BREAKING: Remove the Post and Assessment schemata, re-name UserAccount:number to UserAccount:phone, CryptoWallet:mangingExchange to CryptoWallet:managingExchange, and change the property type of Sanction:duration and License:area to number. Remove Company:ibcRuc, and add hidden Thing:programId. See Resolution plan for divergent types issues opensanctions/followthemoney#5 for more context on type changes.
• New properties: Interval:proof (see Add proof to Interval #1699), Thing:programId (for making sanctioned entities searchably by sanctions policy).
• BREAKING: The method signature for EntityProxy.from_dict() has changed: the first argument, model, is no longer needed. This is a consequence of the model singleton being more widely accessible inside the codebase as Model.instance().
• Introduce a model for dataset metadata. This introduces a Dataset class, which has typical dataset metadata attributes like name, title, summary, coverage (nested), publisher (nested), version, datasets (list of references to child datasets), and a list of resources which describe individual files - often retrievable at a URL, that are underlying the dataset. Multiple datasets can be grouped into DataCatalog, which is essentially a JSON array of datasets.
• Introduce a statement data model for entity data. In this model, the property values of entities are complex objects, which include metadata like lang (language), first_seen/last_seen (freshness), dataset (origin of the statement in the context of merged entities), and original_value (the value of before normalization/cleaning has been applied, e.g. the unparsed date value).
• Remove dependency on fingerprints library. The underlying idea of fingerprints was to make a one-shot attempt at simplifying a given name string into a form that is easy to compare. This has proven to be simplistic in places where we want to apply only some of the heuristics, or where knowing the entity type (person, company) is helpful. This has triggered a new evolution of the tool, which is inlined in rigour as rigour.names.
• Refactor the handling of exports into the RDF format. This used to be spread out across the code base as a first-class concern, but given the relative obscurity of the use case, it made sense to move this into its own exporter module that handles the whole process.
• We've tightened the type signature for various signatures, while keeping in place the code that handles existing inputs. The idea here is to force other Python applications using FtM to start producing more and more clean interface usage based on type hints, which will also result in less ambiguous outcomes and performance gains.
• Remove followthemoney.offshore for tagging specific entities as corp.offshore.

[kdeden]

Can you remind me what this represents?

[kdeden]

tagging @pudo and @tillprochaska for visiblity

[pudo]

The idea of external statements comes in with building an enrichment system. Imagine you have an investigation that you're conducting on a specific domain (place or topic) and you try to expand the entity properties and adjacent entities inside of that investigation with data sourced from much larger, external, data sources.

For example: you're doing a country graph on a relevant jurisdiction and have a dozen datasets you include in full (politicians, legal professionals, registered lobbyists, OC figures, etc. etc.), and another set of sources that provide additional context on a subset of these entities: offshoreleaks, company registries, procurement data, cadastral information, etc.

One way you can build an investigative artifact out of this is by materializing a database of the whole datasets, running a dedupe/xref system on it, and then shipping the deduplicated whole graph. That becomes nasty when the enrichment datasets are very large (EGRUL, PSC), or when you don't actually have bulk data access (e.g. OpenCorporates, OCCRP Aleph, Sayari, in practice also Wikidata).

In those cases, it's more effective to run an incremental pull process in which you query the entities in your investigative graph against the given API and store the possible match results as "external", until an analyst has had a chance to verify the match by running a dedupe process on a version of the graph with external statements included. Once verified/integrated, they become an "internal" entity with the next enrichment run, and become part e.g. of the export to a graph DB used for navigating the investigative graph.

[kdeden]

OK, gotcha. It seems very specific to one use-case, and the kind of thing that should be included in a specific implementation, but that said I'm not opposed to including it if it's going to cause headaches.
As we discussed, we're thinking aloud about ways to make the statement dict extensible while without gobbling memory. Maybe external could live there for your use-case?

[kdeden]

@pudo this is a 1:1 mapping with prop, right? So if you know the prop then you know the prop_type? Do we need to keep it?

[tillprochaska]

Had a similar question, the one use case where I could imagine storing the type separately is to build some kind of migration tool when the type of a property changes? Are there other cases (where deriving a list of properties from the model isn’t enough)?

[pudo]

No, we've just been including it in our exports for convenience ("give me all statements of type 'name'" kind of stuff, see eg. https://github.com/opensanctions/rigour/blob/main/contrib/address_tokens.py#L14

But I agree that it's probably cleaner to do this in our output formatters and drop it from the spec. I can make that into a PR

[tillprochaska]

Note to myself/for future reference:

In Aleph, an entity’s fingerprints are materialized to the ES search index, primarily for blocking/querying for candidates during cross-referencing. I was wondering whether the change from fingerprints to rigour.names could be problematic regarding backwards compatibility with an "old" index.

When generating cross-ref candidates for a given entity, Aleph computes fingerprints based from the entity on the fly (using fingerprints directly) and constructs an ES query based on that. This means that fingerprints at index time and query time are still generated using the same "old" approach from fingerprints, so that shouldn’t be a problem (except for not making use of the newer rigour approach).

• Indexing: https://github.com/alephdata/aleph/blob/dd572d55bdf235af01e5986392d2890daef323b3/aleph/index/entities.py#L196
• Querying: https://github.com/alephdata/aleph/blob/dd572d55bdf235af01e5986392d2890daef323b3/aleph/logic/matching.py#L26

[tillprochaska]

Aleph also uses followthemoney.compare for the entities/<id>/similar API endpoint:

• First, it fetches candidates (in the same way as during cross-ref), i.e. again the same "old" fingerprints approach will be used and there shouldn’t be a problem.

• Then it compares the given entity with each candidate to compute a similarity score. Here, it’s indirectly using the new approach from rigour.names via followthemoney.compare for both the entity and the candidate, so this also shouldn’t be a problem.

While the blocking/candidate generation step would be using a slightly different approach compared to the scoring step, I can’t imagine this being a problem given that the underlying idea is still the same (heavy normalization of names to a canonical form).

https://github.com/alephdata/aleph/blob/dd572d55bdf235af01e5986392d2890daef323b3/aleph/views/entities_api.py#L374

[tillprochaska]

To summarize: These changes shouldn’t require significant changes in Aleph. That said, it does of course make sense to align the normalization and eventually discontinue use of fingerprints in Aleph as well, and if it’s just to reduce legacy code and complexity.

[pudo]

Your concern is totally justified, but has always been an ugly secret of fingerprints: it was never a very stable function. Not really meant to be, either, it has too much moving world knowledge inside of it. And it depends on stuff like pyicu and normality that are also moving. So the fingerprints in the Aleph index was always a crapshoot - a thing that works best if the index is a bit short-lived.

[tillprochaska]

That’s fair enough and good context, but at least I now know the change to rigour.names won’t make this issue worse :)

[tillprochaska]

Is this necessary? License already inherits address and addressEntity from Thing.

[pudo]

You're totally right. Let's kick it out.

[tillprochaska]

@pudo I know you wanted to remove the IBAN type before in favor of identifier with an IBAN format. Maybe 4.0 is the right time to do that (even though that will require adjustments in Aleph)?

[pudo]

That would be amazing. I imagine it actually requires adaptations in ingestors and even the UI (dataset home screen).

[tillprochaska]

I think ingest-file should be fine, mostly, except that we might want to align the IBAN parsing. Looking at Aleph, indexing/cross-ref, stats/UI, and searching/querying are probably the main areas that would need to be adjusted. I could imagine that e.g. having a list of top IBANs is still quite useful, so we might want to compute stats for IBANs even though they aren’t a separate type anymore.

[tillprochaska]

Also curious what @brrttwrks @catileptic @simonwoerpel think about this?

[pudo]

FWIW there's a concept of "strong identifiers" now, too, which could be an interesting thing to show on the home page. It's stuff like IBANs, LEIs, OGRNs - stuff that can be checksummed and is pretty clearly meant to identify one entity....

[catileptic]

Validating IBAN feels very "best-effort": throwback to when I compared two Python libs that attempt this. I would be comfortable removing it.

[tillprochaska]

FWIW there's a concept of "strong identifiers" no

That’s only in rigour for now, right?

[tillprochaska]

Suggested change

	For example, this could be useddocker to say: "In dataset A, entity X has the
	For example, this could be used to say: "In dataset A, entity X has the

[tillprochaska]

I’m probably missing something here, but I don’t understand why it’s necessary to override this?

• StatementEntity is an EntityProxy and inherits the caption getter. As StatementEntity overrides get, shouldn’t that work out of the box?
• The only difference between pick_caption and the caption getter from EntityProxy seems to be that the former caches the result and uses pick_name from rigour for name props.

[tillprochaska]

So to state my question explicitly: Is there a reason why we can’t have the caption getters share the same implementation (maybe except for the caching part if sensible)?

[pudo]

No. I think medium-term if may be interesting to pick_caption to take tuples of (name, lang?) so that we can pre-select english names when that metadata is available, but that can still be one implementation.

[tillprochaska]

I was wondering whether it would make sense to make this generic and leave some of these attributes to application implementations, in particular things like first_seen, last_seen, external, lang?

They might not be relevant for all use cases, and I can imagine that many use cases require additional attributes and thus will have specialized Statement classes anyway? For example storing whether a property was provided by an end user/a specific crawler/ingestion stage (unless you model that as separate datasets), application version, user ID, versions for time travel, …

(I do have to admit though that I’m not sure how much complexity this would add given that some of the logic for reading/writing and, in case of external, generating IDs, hard-codes these attributes.)

[kdeden]

Great. cc @pudo
What would making a Generic look like in usage? You import ftm and then define a generic, then call some method on CompositeEntity to bind it?
Alternatively, we could define additional statement attributes in a YAML, and use type() to construct it with __slots__. What do you think?

[pudo]

Yeah that's how I implemented it in nomenklatura initially (taking the theme of the E and SE typevars). The problem was that it blew up the complexity of the code in lots of places (you end up with generic generics), and in some cases lead me into open type issues that I couldn't really resolve.

The broader point I'd make is: part of the objective here is to evolve FtM as an interchange format as much as an internal domain model. Making the core element of it have a "proprietary specials" section sort of abandons that objective a bit. Do we want that? It's basically a bit like turning FtM into a data modeling language rather than an actual data model.

If OCCRP is institutionally unable to share their requirements: It's also conceivable we just end up implementing two flavors of statements - open-source statements, and OCCRP statements. Would that perhaps take the load off this?

I sort of feel it's hard to design against an unknown set of requirements...

[kdeden]

Can you explain a bit more about the problems you've experienced with the complexity of the code? I agree Python's typing system is unwieldy in situations like these, but I also know that the current precedent for addressing this kind of problem is to have model reading from YAML and having an EntityProxy class to operate on records.

If I understand correctly, you're saying that extending this logic into the structure of the statements themselves is paradigmatically different ('turning FTM into a data modeling language rather than an actual data model'). I think our goal (everyone's goal) is to keep this as a data model, but make sure it's extensible, and not tell people what they ought to do with it.

On that note, FtM consumers will want to store other information that is related to their use-case on a statement level, (e.g. as you have with external and canonical_id). I think that's in line with the general vision, and I'm mainly interested in how to implement this technically.

re desigining against an unknown set of requirements -- OCCRP has a good idea of what we'd want out of a statements-as-rows database model (cc @gondzo, but it's not something we'd want to enforce on others. If it's technically feasible, I prefer to decide on a ballpark domain of database logic FTM should support and find a way to leave that mostly open.

If it's not technically feasible, the other option would be to pack the Statement model with all of the potentially attributes we can think of now, and continue updating as the need arises from the community. (e.g. is_ai_generated, version, fact_checked).

Anyway, can you share a bit more about those limitations you encountered? How did you get to generics of generics?

[pudo]

The thing that I felt about generic statements was that they become very concrete in every single place you use them: the serialization/deserialization code needs to be different, any sort of database storage is different, the way you'd expose them through the statement proxy is different. The place where I ran into real type trouble was around trying to describe stuff like an abstract graph data backend (for context: CE in that code is SE in this PR), which then becomes generic on like two levels (CE is generic over S). Then you get to do all this in multiple languages.

I'm aware that I'm coming off super aggressive/defensive in discussing the statements stuff, but part of the reason for that is the realization (painfully gathered over the last four years) that making statement data work in prod is a hellish bit of work. We've churned through Postgres, BigQuery, Redis, DuckDB as backends, and nothing works well. That's sort of what makes me think that in order to get this to work, we should either make one structure that is super opinionated, or - hear me out - two: one that's basic in descriptiveness but works for large datasets (like EGRUL), and one that has full "properties for properties" support (wikidata style), but does not have the goal of processing bulk data (instead focus on smaller datasets). Wikidata is a good case study of this: loading scientific citation data basically broke their database in terms of scale, so now they're trying to avoid bulk data imports. That same data, when formatted as FtM is basically mid-size for the sort of material we're sometimes dealing with.

[tillprochaska]

The broader point I'd make is: part of the objective here is to evolve FtM as an interchange format as much as an internal domain model. Making the core element of it have a "proprietary specials" section sort of abandons that objective a bit. Do we want that?

Thanks for clarifying and for sharing your experience regarding the complexity of a generic implementation, that makes sense. Also agree that the statement model here doesn’t need to map 1:1 to internal storage, and this resolves some of the use cases I mentioned above. For example, the ID of the user who added a property value can simply be dropped when serializing to statements. This probably also applies to stuff like the version of FtM or ingest pipeline used which isn’t useful beyond system boundaries.

There are other use cases where that decision isn’t 100% obvious, for example: In Aleph’s current fragment model, each fragment has an origin which can be used to designate that the fragment originated from a certain stage of the ingest pipeline, a particular scraper, an ML/AI model, a user, … This might be useful to keep when sharing statements, but maybe it’s also fine to just drop that information.

Then there are some attributes for specific use cases (such as fact_checked as mentioned @kdeden above. @kdeden What do you think about the third option of treating these as internal implementation details for now, then potentially upstreaming them later?

[kdeden]

OK, I'm back from vacation. I know we all want to get this merged and done.

I propose a compromise situation where we include, in 4.0, the following members included thus far in the PR:

1. id: Optional[str]
2. entity_id: str
3. schema: str
4. prop: str
5. value: str
6. dataset: str
7. lang: Optional[str]
8. original_value: Optional[str]
9. external: bool
10. canonical_id: str

These additional members that are present in followthemoney-store:

11. origin: Optional[str]
12. fragment: Optional[str]

And an extras dict that can hold whatever the caller of either Statement constructor wants:
13. extra : Optional[dict[str,str|int|bool]](extra: Optional[Dict[str, str | int | float | bool | None]]

• init and from_dict will take these keys into account, init will raise a valueerror if any of the extra keys overlap with the Statement members.
• This setup ensures that nothing in the extras dict can be used in any of the internal statement logic, e.g __eq__, generate_key, make_key, etc. It's just there for the user to store because it sparks joy.
• to_dict remains as it is -- it returns a StatementDict object. Methods like to_db_row and to_csv_row must be modified to include the key/value pairs from the extras dict, (in a flat structure, not as a subordinate object).

@pudo Do you feel this reflects what we talked about?

[tillprochaska]

Another use case for null property values (but this doesn’t have to be addressed now):

A common feature request for Aleph is to store invalid values. Sometimes even invalid values are relevant data points, but in other cases it’s simply helpful for debugging or QA reporting.

This use case would be covered by null property values (in combination with original_value).

[pudo]

OK, I've just adopted most of @tillprochaska's feedback:

• Remove License:location
• Remove registry.iban
• Remove StatementEntity.caption (or rather: move it upstream)
• Remove Statement.prop_type (it's there as a property, and in the CSV export format)

[kdeden]

I notice that this works perfectly well with the old fashioned EntityProxy. Is there an issue with expanding this signature to include that? Seems like a great compatibility feature.

Suggested change

	dataset: str,
	entity: "EntityProxy",

[kdeden]

Example:

person: EntityProxy = model.make_entity("Person")
person.make_id("test_person")
person.add("name", "John Doe")
person.add("birthDate", "1980-01-01")
person.add("email", "john.doe@example.com")

for statement in Statement.from_entity(person, dataset="test"):
    print(statement.to_dict())

=====OUTPUT=====
{'canonical_id': 'f73e8656f8a7a531b33d474fc54f9759e3343bf5', 'entity_id': 'f73e8656f8a7a531b33d474fc54f9759e3343bf5', 'prop': 'id', 'schema': 'Person', 'value': 'f73e8656f8a7a531b33d474fc54f9759e3343bf5', 'dataset': 'test', 'lang': None, 'original_value': None, 'first_seen': None, 'last_seen': None, 'external': False, 'id': '9385d7840a5a553d58a5a5e10491b9d56cdc35e8'}
{'canonical_id': 'f73e8656f8a7a531b33d474fc54f9759e3343bf5', 'entity_id': 'f73e8656f8a7a531b33d474fc54f9759e3343bf5', 'prop': 'name', 'schema': 'Person', 'value': 'John Doe', 'dataset': 'test', 'lang': None, 'original_value': None, 'first_seen': None, 'last_seen': None, 'external': False, 'id': 'ab5a7ceab0ea9342546f2acaa18d9e9605328303'}
{'canonical_id': 'f73e8656f8a7a531b33d474fc54f9759e3343bf5', 'entity_id': 'f73e8656f8a7a531b33d474fc54f9759e3343bf5', 'prop': 'birthDate', 'schema': 'Person', 'value': '1980-01-01', 'dataset': 'test', 'lang': None, 'original_value': None, 'first_seen': None, 'last_seen': None, 'external': False, 'id': 'c7afdd693fe54b30ae043a962b84ba1e10b94fe9'}
{'canonical_id': 'f73e8656f8a7a531b33d474fc54f9759e3343bf5', 'entity_id': 'f73e8656f8a7a531b33d474fc54f9759e3343bf5', 'prop': 'email', 'schema': 'Person', 'value': 'john.doe@example.com', 'dataset': 'test', 'lang': None, 'original_value': None, 'first_seen': None, 'last_seen': None, 'external': False, 'id': '66165c4fbf27beb339b72141669cce7a4edddede'}

No data is added, but the format is compatible.

[kdeden]

Actually, it doesn't seem to work in the expected way for the StatementsEntity, because the members of Statement are set without regard for the input values.

e.g.

    person: StatementEntity = StatementEntity(
        Dataset.make({"name": "testdataset"}),
        {"schema": "Person", "properties": {}},
    )
    person.id = "test_id"
    person.add_statement(
        Statement(
            entity_id="test_id",
            prop="education",
            schema="Person",
            value="University of California, Los Angeles",
            dataset="testdataset",
            first_seen="2025-01-01",
            last_seen="2025-01-01",
        ),
    )
    person.add_statement(
        Statement(
            entity_id="test_id",
            prop="education",
            schema="Person",
            value="University of California, Los Angeles",
            dataset="testdataset",
            last_seen="2025-01-02",
        ),
    )
    print("--------------------------------")
    for s in Statement.from_entity(person, dataset="testdataset"):
        print(s.to_db_row())

returns

{
    'canonical_id': 'test_id',
    'entity_id': 'test_id',
    'prop': 'id',
    'schema': 'Person',
    'value': 'test_id',
    'dataset': 'testdataset',
    'lang': None,
    'original_value': None,
    'first_seen': None,
    'last_seen': None,
    'external': False,
    'id': '54cbed65f3a51c2c859014bebd91529f0a77557b'
}
{
    'canonical_id': 'test_id',
    'entity_id': 'test_id',
    'prop': 'education',
    'schema': 'Person',
    'value': 'University of California, Los Angeles',
    'dataset': 'testdataset',
    'lang': None,
    'original_value': None,
    'first_seen': None,
    'last_seen': None,
    'external': False,
    'id': '2509f49fd78932551c199f1b9f47a91aafa40b22'
}

So none of the data from the Statements other than the value has propagated. Am I missing something here @pudo?

[kdeden]

@pudo
I can modify this to use EntityProxy instead, which basically 'upgrades' an FTM <4 statement to 4.0, is that the intended usecase?
Or it can use StatementEntity and in that case, I feel like the statement data should propagate down, unless I'm missing something? Just lmk

[kdeden]

In my conversations with data journos and people producing ftm data, we would love a class that we can use as an EntityProxy but with Statements, and use it in the same way to produce FTM data:

entity = model.make_statement_entity("Person")
entity.make_id("test_person")
entity.dataset = "testdataset"
entity.add("name", value="John Doe", last_seen="2025-01-01", origin="some process")
entity.add(
    "education",
    value="Universidad de Santiago de Chile",
    last_seen="2025-01-01",
    lang="es",
    # no need to specify the dataset/schema/entity_id, it will be added automatically
)

Could that be this class? It seems like this class is only meant to be used by FTM consumers. Is that intentional?

[kdeden]

Suggested change

	origin: Optional[str]
	fragment: Optional[str]
	extra: Optional[Dict[str, str | int | float | bool | None]]

[kdeden]

Suggested change

	"last_seen",
	]
	"last_seen",
	"origin",
	"extra",

[pudo]

Not to be a pest, but: what's the meaning of fragment, here? To me, the charme of statements is that they get rid of fragments (both mechanisms do the same thing, but statements do it a bit more elegantly).

[kdeden]

Suggested change

	):
	self.entity_id = entity_id
	origin: Optional[str] = None,
	fragment: Optional[str] = None,
	extra: Optional[Dict[str, str | int | float | bool | None]] = None,
	):
	if extra is not None:
	for key in extra.keys():
	if key in self.__slots__:
	raise ValueError(f"Key {key} is a reserved slot")
	self.extra = extra
	self.origin = origin
	self.fragment = fragment

[kdeden]

Suggested change

	def from_dict(cls, data: StatementDict) -> "Statement":
	return cls(
	entity_id=data["entity_id"],
	prop=data["prop"],
	schema=data["schema"],
	value=data["value"],
	dataset=data["dataset"],
	lang=data.get("lang", None),
	original_value=data.get("original_value", None),
	first_seen=data.get("first_seen", None),
	external=data.get("external", False),
	id=data.get("id", None),
	canonical_id=data.get("canonical_id", None),
	last_seen=data.get("last_seen", None),
	)
	def from_dict(cls, data: StatementDict) -> "Statement":
	data_copy: Dict[str, Any] = dict(data)
	return cls(
	entity_id=data_copy.pop("entity_id"),
	prop=data_copy.pop("prop"),
	schema=data_copy.pop("schema"),
	value=data_copy.pop("value"),
	dataset=data_copy.pop("dataset"),
	lang=data_copy.pop("lang", None),
	original_value=data_copy.pop("original_value", None),
	first_seen=data_copy.pop("first_seen", None),
	external=data_copy.pop("external", False),
	id=data_copy.pop("id", None),
	canonical_id=data_copy.pop("canonical_id", None),
	last_seen=data_copy.pop("last_seen", None),
	origin=data_copy.pop("origin", None),
	fragment=data_copy.pop("fragment", None),
	extra=data_copy or None,
	)

[catileptic]

@kdeden I'm looking at refactoring OpenAleph to use FTM 4.0. I was wondering how much work you foresee this PR will still take? A very rough estimate is absolutely fine :)

[kdeden]

Thanks for the nudge @catileptic . I anticipate it done this week, by Friday. It's really a matter of
a) adding all the convenience methods and constructors and making sure this new Statement, StatementDict and StatementEntity function as expected.
b) maybe adding another test for serializing and deserializing statements
Maybe @pudo and I can get through it in one session tomorrow?

[kdeden]

Nothing else that warrants a major version. Apologies to @pudo for this dragging on, and many thanks to all.
I see some commits in this long PR are not signed, can the guilty parties correct or can @stchris override if he's OK with that?

[stchris]

I'm OK with that but: Base branch was modified. Review and try the merge again.

[kdeden]

• I rebased this branch to incorporate the latest changes from the main branch. If you have anything last minute to add (unlikely) you may need to resync.
• The docs build failing is due to hyperlink finding broken links, although the broken links are all internal and directory structure hasn't changed. I will investigate this again in a few hours.