No uniqueness constraint on entity names makes name-keyed upsert silently racy

[bharkema]

A finding from our translation layer that I think applies to anyone writing to STA concurrently, and that produced wrong data with no error of any kind.

The pattern. Server-side ids are server-generated, so a client that has to be restart-safe cannot store them as its key. The advice in #11 applies: resolve entities by querying for them. That means every writer implements the same upsert: GET ?$filter=name eq '', then POST if absent.

The problem. That is check-then-act, and STA does not require entity names to be unique. So when two requests race, both find the entity absent and both create it, and the server is right not to complain. We resolve datastreams in parallel, and entities that are shared between datastreams (one Sensor per sensor model, one ObservedProperty per phenomenon) are exactly the ones that race. We ended up with five copies each of one Sensor and one ObservedProperty.

Why this is worth raising rather than just fixing. No 409, no error, no signal. We only noticed because a later name lookup returned five hits where it expected one, and we happened to have written that lookup to log multiplicity. Implemented the obvious way, taking the first result, this stays invisible forever. Data is silently duplicated and every downstream consumer sees a plausible answer.

We fixed it on our side with a per-key single-flight around the create, so concurrent first-sight resolutions of one name collapse into one lookup and one POST. That works, but it is a client-side fix for something the server could settle definitively.

What would remove the class of bug, roughly in order of preference in our opinion:

• A standard idempotent create: a client-supplied natural key, or an If-None-Match-style conditional POST, returning the existing entity instead of a duplicate.
• A way for a server to declare a uniqueness constraint (e.g. Sensor.name is unique) and return 409 on violation. Even just this turns silent corruption into a loud error a client can converge on.
• Failing both, a documented upsert-by-name pattern in the spec, so implementers at least hit the race knowingly.

Related, and a reason not to solve it with names alone: on a shared server, name-keyed upsert also means our lookup can bind to another participant's entity that happens to share a name, inheriting their definition and units silently. We prefix every entity name we create to avoid it. That works, but it guarantees fragmentation: if two parties genuinely should converge on one Fill level concept, prefixing ensures they never do. Namespacing on shared servers might be worth a convention at programme level.

Has anyone else writing concurrently hit this? @hylkevds, is client-defined id / uniqueness something v2 addresses?

[hylkevds]

In STA v2 we've added:

In most cases the value of the id attribute of an Entity will be generated by the service when the Entity is created, but a service may allow users to specify a value.

The problem with uniqueness is a tricky one. The standard can't say too much about it, since what makes an Entity unique depends very much on the use case. Two Things can both have a Datastream with the same name. Two Things may have the same name, but a different Location. Having multiple Observations in the same Datastream with the same phenomenonTime is quite common in the water domain, where a lab analysis is often repeated with different parameters...
A server admin is free to set additional constraints in the database to ensure uniqueness, after discussion with the local domain experts. This can also include any attributes in the properties Object. We could also have done that for this project.

When importing data I always have a single thread handling non-Observation entities, and a per-datastream thread handling Observations. Since inserting an Observation requires a lock on the Datastream, there is no performance benefit to having multiple threads inserting Observations for the same Datastream. Instead, I use data arrays or batch-processing to batch-upload Observations.
Since there is only one thread for a Datastream, that thread only needs to check once what the latest Observation in a Datastream is. Once it fetched that information, it knows that any Observation with a later time will not exist yet, and can safely be created.

[bharkema]

Our natural-key claim generalised from the deployments we had seen. We wrote that a datastream plus a timestamp identify an observation "in every real deployment", and your lab-analysis example is the case we did not have: repeated analyses of one sample, sharing a phenomenonTime but differing in parameters. That shape is legitimate. Mandatory dedupe on a natural key would therefore not be a safety improvement: it would break a valid use case. We are correcting the claim in our report rather than narrowing it.

What survives is narrower, and we think still worth something. Not "the standard should enforce uniqueness", which your three counter-examples rule out, but two separate points: a client should be able to find out what a given server enforces, and a client should have a portable way to create without reading first. Both are independent of uniqueness, and the rest of this reply is about them.

Client-specified ids in v2 answer our entity-level problem. If we can supply the id, our upsert stops being check-then-act at all. We derive the id deterministically from the vendor key, write at that id, and the primary key does the deduplication that no amount of client-side locking can guarantee. That removes a round-trip per entity as well as the race. Three questions about it:

1. Is it discoverable? "A service may allow users to specify a value" is exactly the kind of optional behaviour a portable client currently has to establish by trial. Is there a conformance class for it in v2, or something in the landing document that answers yes or no?
2. Is there create-or-replace at a known id, so a client can be idempotent in a single request instead of treating the "already exists" error as a normal path?
3. How does this relate to the definition field you raised in UUID's - First Class Optional Field? #12 for external references? If we already hold a stable vendor key, is the recommendation to put it in the id, or to keep server-generated ids and carry the vendor key in definition?

Server-side constraints relocate our recommendation rather than remove it, and we think in a useful direction. You are right that an admin can add constraints after consulting domain experts, properties included, and that this project could have done so. On the shared testbed instance, a unique constraint on Sensor.name and ObservedProperty.name would have turned our silent duplication into a 409 on the first run, and we would support adding one if others agree.

The wider point we take from that: on a server that several parties write into, the set of constraints in force is a deployment decision that every writer depends on and none of them can see. That is not a gap in the specification. It is a gap in what a national profile should pin down, and we are moving it there in our report. It is also a second argument for a machine-readable capabilities document, since admin-added constraints are exactly the kind of local truth a client cannot guess.

Your import architecture is a better answer than the fix we had planned. We probe once per observation. You probe once per datastream and then rely on the ordering. We had already noted that our stored cursor makes the probe redundant, but we had not connected that to threading: if inserting an Observation locks the Datastream, per-datastream parallelism buys nothing and the single-threaded design costs nothing. That also explains something we had recorded as a puzzle, namely that adding concurrency did not move our throughput. We are adopting the pattern: one thread per datastream, one latest-observation query per run, then batch upload.

One case still needs the write log alongside it. Your invariant is that nothing later than the newest stored observation can exist yet, which holds when a source only ever moves forward. Ours does not always. The vendor API mixes settled estimates with forecasts, and it can re-issue a corrected value for a timestamp we have already passed. For those, "later than the maximum" does not help, and we still need per-key bookkeeping. Do back-dated corrections come up in your imports, and if so, do you handle them in a separate reconciliation pass rather than inside the forward-only path?

[mathisvfr]

Practical update from our side.

We adopted the per-datastream threading model @hylkevds described, and it removed both the race and the performance puzzle we had been chasing: adding concurrency to observation inserts was not helping because each insert locks the Datastream anyway. One thread per datastream, one latest-observation query at startup, then batch upload. Simpler and faster.

On the shared-server namespacing point: we prefix every entity name with GEO_ and bind entities to a Project per site (GEO_TGV, GEO_BDJ). It works for isolation, but it does guarantee that two parties who genuinely model the same phenomenon will never converge on one entity. That trade-off feels like something a national profile should settle, not each implementer independently.

One thing we can now confirm from running three targets concurrently: the write-capability discovery step we built (does this server support dataArray batch? client-specified IDs? what does it do on duplicate?) has been essential. Each of the three servers (FROST v1.1, CollaborAll v1.1, Fraunhofer v2.0) behaves differently, and there is currently no way to discover that programmatically. A machine-readable capabilities document, even a minimal one, would remove a lot of trial-and-error.

Still very interested in @hylkevds's answers to the three v2 questions above :)

[hylkevds]

SensorThings has a machine readable capabilities document since version 1.1. It's on the lading page in the serverSettings/conformance section. Of course that only covers the things that have a conformance class written for it, such as if data-arrays or batch-requests are supported. Though a server can also add its own conformance class that describes the business logic of the server.
How IDs are generated is something that should go in the V2.0 data model description. It's quite likely that there already exists an annotation for that.

The duplications can (almost) be avoided using a batch-request with a get followed by a conditional insert, which is something we plan on adding to the upcoming extensions. Currently FROST does it like: https://fraunhoferiosb.github.io/FROST-Server/extensions/JsonBatchRequest.html#back-references-to-get-requests
While a race condition can still happen, it is much less likely to happen inside a single request, than if there is an HTTP round-trip between the GET and POST.