[implemented] id capture

[matko]

We've discussed this a little bit before but let's register a discussion item for it.

For some document inserts it'd be really useful if we were able to point back to an earlier inserted document, or even to ourselves. Right now, we can't really do that, requiring you to go in a second time and update existing records after you know the ids. Depending on the constraints, this is not always possible.

What we kinda should be able to do is something like this:

{
    "@type": "Person",
    "@id": "@capture(mike)",
    "name" : "Mike",
    "birthdate": "1970-03-12"
}

{
    "@type": "Person",
    "name" : "Bob",
    "birthdate":"2000-01-02",
    "father" : "@capture(mike)"
}

This should allow us to refer back to any ids of earlier inserted documents.

That still leaves loops though. Sometimes you want to refer to something that will be inserted later, or maybe even the exact document you're inserting right now. This scheme would not support it.

For self-referral we could perhaps support a special @capture(self) or just @self. This should be usable anywhere except in any of the key fields, though it'll complicate elaboration a little bit.

For forward references, I still think it should be possible to also refer to things through their key fields, allowing any order of insertion:

{
    "@type": "Person",
    "name" : "Bob",
    "birthdate":"2000-01-02",
    "father" : {"name": "Mike"}
}

{
    "@type": "Person",
    "name" : "Mike",
    "birthdate": "1970-03-12"
}

This is currently broken however.

[matko]

Forward references through captures should actually be possible too in some situations, but they'll complicate the process. We'll have to reorder operations or do some coroutining magic to ensure that we know these ids before we insert something that needs to know about this id.

There are however situations where this would never resolve right. When id generation depends on a field for which the id has yet to be determined, it's possible to create a deadlock where a group of objects is not able to get an id because there's a dependency cycle. We can probably detect this though, and throw an error for that case.

[spl]

Can we use something besides @capture? Perhaps @ref or @refid?

[spl]

Look into how JSON Schema does it.

[matko]

Reviving this topic as I'm looking into this again.

I'll just use this space to write down my notes on how I think this is to be implemented:

documents can specify '@capture' as an extra field, the value of the field will be a string serving as a capture variable name.

whenever an id is referenced, documents can then specify a dictionary like
{ "@ref": "capture_name"}

to specify that they wish to use that id

so how to make that work?

we expand json_elaborate with three extra arguments:

• +Capture_Map: an association list containing all capture variables up to now
• -New_Capture_Map: an association list containing the capture variables known after running this elaboration
• -Capture_Dependencies: A list of variables that need to be ground before this document can be inserted

json_elaborate can then use these maps as follows:

• whenever '@capture' is encountered, the map can be expanded with a pair "capture_name"-Id
• whenever '@ref' is encountered, the dict it is in can be replaced by a variable
  -- if the capture_name is in the association list, this variable can be unified by the right hand side of the pair
  -- if not, a new entry is to be added to the association list

insert_document_expanded should then get an additional variable, namely the list of captures that need to be ground. It can then use when(ground(Dependencies), ...) to delay insertion until these elements are ground.

insert_document and replace_document should get the same extra arguments as json_elaborate, to pass them through to json_elaborate.

Whoever then calls insert_document or replace_document should set up the association list, and after inserting/replacing everything, ensure that the full association list is ground. If any element is not ground, that is an error which should be reported to the user ('id reference "foo" did not have a corresponding id capture in any docuemnt').

Finally, currently the api greedily prints ids for insertion and replacement, but since insertion may be delayed, or due to a missing capture, not happen at all, we can't do that anymore. this code will need to be rewritten to collect the ids and then print them at the very end.

[matko]

This has now been implemented.