Only writes on stop document

[EliotGann]

Not sure if this is fundamental to the structure of json metadata suitcase, but is there any reason why it can't partially write the json on the start document, so that this metadata can be used for live analysis? Right now it doesn't even create the file until the stop document.

[tacaswell]

There is not an append-only way to extend a json file (as you have to insert new next before the closing }) .

If there is no option but to do communication to the live analysis via files, I guess we need a suitcase-jsonl-metadata as well. Would it be possible to have the live analysis look at a zmq socket instead?

[EliotGann]

I don't know if there is anything unique in the stop document that isn't (or can't be) in the start already, is there? it's not like the metadata is changing during the scan?

I don't know much about ZMQ, but it seems like it could be a possibility. (one that would require a bit of coding on the analysis side, which I was hoping to minimize while I rewrite the analysis in a python friendly way)

[danielballan]

I don't know if there is anything unique in the stop document that isn't (or can't be) in the start already, is there? it's not like the metadata is changing during the scan?

I'm not sure what you mean. The stop document contains information that we can't know at the beginning, like how it exited (success/failure) and when it exited.

What code is consuming this json-metadata file---Igor?

[EliotGann]

From a live analysis point of view the only interesting info in the Json to me is the contents of the md dictionary. This can’t be used for live plotting or analysis (in Igor) when it isn’t written until the whole scan is complete. A different suitcase that just wrote the md dictionary on the start document and that was it, I would be happy. I would have called this json metadata, but I’m probably confused about the scope of what json metadata is meant to cover.

[danielballan]

Oh I see. I think this misunderstanding validates a concern we had about inventing user-facing formats like "json-metadata" without a specific consumer in mind.

You can bend suitcase-jsonl to your purposes like so:

# Write a JSON file with just the RunStart document in it
# as soon as the document is emitted by the RunEngine.
import event_model
import suitcase.jsonl

def factory(name, start_doc):
    with suitcase.jsonl.Serializer(directory='...', file_prefix='...') as serializer:
        serializer(name, start_doc)
    # We do not need any further documents from the RunRouter.
    return [], []

rr = event_model.RunRouter([factory])
RE.subscribe(rr)

Tested interactively, works.

[EliotGann]

Excellent. Thank you!

[danielballan]

Let's make the action item on this issue "improve the documentation." We already decided that we won't bother to do full sphinx-based docs for every suitcase. I propose just adding some information to the README.

@awalter-bnl Can you add some rough documentation to the README that shows the example output file for a basic example like RE(count([det]))? And maybe also copy my example above for "If you just want the start document..."?

[awalter-bnl]

I will start that now @danielballan

[awalter-bnl]

Issue moved to bluesky/suitcase-jsonl #14 via ZenHub

[awalter-bnl]

moved this issue as it was resolved via a change to the readme file in suitcase-jsonl

[danielballan]

Documentation for suitcase-jsonl is welcome, but the problem in this issue was confusion around the intent of suitcase.json_metadata and the structure of its output, not suitcase.jsonl. I only involved suitcase.jsonl to write a workaround. Can you also write a README for suitcase-json-metadata with an example?

[danielballan]

Hm. I see that "moved by ZenHub" creates a copy of all the comments whereas GitHub's "Transfer this Issue" is better because it actually moves the issue instead of copying.