Doc issue 78 #94

jklynch · 2019-06-29T01:30:39Z

Move event-model documentation from bluesky to event-model.

Description

Merge https://blueskyproject.io/bluesky/documents.html into https://blueskyproject.io/event-model/data-model.html.

Move https://nsls-ii.github.io/bluesky/event_descriptors.html into the event-model project.

Motivation and Context

This PR addresses issue #78.

How Has This Been Tested?

jklynch · 2019-07-01T19:22:40Z

I left some text between "begin remove" ... "end remove" markers.

danielballan · 2019-07-02T13:31:17Z

docs/source/data-model.rst

-Data Model
-**********
+********************************
+Data Model: Documents and Events


I think of Events as one of the kinds of Documents.

I have had some confusion with the event/Event terminology. In my mind there are event-based execution model "events" and then there are "Event Documents". I try not to think of the documents as "events" but as "Events". I think you are right, this heading should go back to just "Data Model".

So, let's be clearer. Instead of starting with the next paragraph, start with something more direct.

In data acquisition, an *event* is a set of readings coordinated at a specific time. A primary design goal of bluesky is to enable better research by recording rich metadata alongside the event data for use in later analysis. Bluesky encapsulates all information from each event in a *Document*. Other types of Document describe the events associated with an experimental run.

danielballan · 2019-07-02T13:32:00Z

docs/source/data-model.rst

+
+A *document* is our term for a Python dictionary with a schema --- that is,
+organized in a
+`formally specified <https://github.com/NSLS-II/event-model>`_ way --- created


This link is a cross-reference from bluesky to event-model. Now that we are in event-model, I think we can just remove this link.

AND, leave the title of this section as "Data Model: Documents and Events" since it creates the necessary association with the data acquisition term event and the bluesky document type event.

danielballan · 2019-07-02T13:37:47Z

docs/source/data-model.rst

 Minimal nontrivial valid example:

 .. code-block:: python

+   # 'stop' document


Do you think we should add a comment like this to all the examples? Most of them do not have one. I would rather apply it to none or all.

I added them to all the examples.

danielballan · 2019-07-02T13:39:10Z

docs/source/data-model.rst

+Data Model: Documents and Events
+********************************
+
+*begin remove*


I still like this original first paragraph, doesn't seem exactly redundant to what follows. Maybe combine it with the next paragraph?

I spent some time trying to combine the original first paragraph and the text from documents.rst. My problem is that the original is talking about little-e "events" and it seems confusing to talk about big-e "Events" at the same time.

I will try again.

Here is a new proposal: I added the original paragraph to the second new paragraph. I still think using the term "event-based" is confusing so I propose using "document-based" instead. I felt like the original text fits here since the following paragraph talks about subscribing to a stream of documents.

A primary design goal of bluesky is to enable better research by recording rich metadata alongside measured data for use in later analysis. Documents are how we do this. A *document* is our term for a Python dictionary with a schema created by the RunEngine during plan execution. All of the metadata and data generated by executing the plan is organized into documents. Bluesky's document-based data model supports complex, asynchronous data collection and enables sophisticated live, prompt, streaming, and *post-facto* data analysis. A :doc:"later section <callbacks>" describes how outside functions can "subscribe" to a stream of these documents, visualizing, processing, or saving them. This section provides an outline of documents themselves, aiming to give a sense of the structure and familiarity with useful components.

working on the third paragraph below....

Hmm, I agree this is tricky. This line once made sense

created by the RunEngine during plan execution

but is now too limiting: we can "ingest" data files from non-bluesky beamlines or simulations to create documents now. I wouldn't want to give the wrong impression that the RunEngine is the only way to mint documents.

danielballan · 2019-07-02T13:40:08Z

docs/source/data-model.rst

+by the RunEngine during plan execution.  All of the metadata and data generated
+by executing the plan is organized into documents.
+
+A :doc:"later section <callbacks>" describes how outside functions can


This is one of the bluesky internal references of the kind I think we should just remove. Maybe this whole paragraph can go?

I liked the mention of subscribing to a document stream. It convinces me that this is very useful information. I propose to link to the Bluesky documentation:

The `bluesky documentation <https://blueskyproject.io/bluesky/>`_ describes how outside functions can "subscribe" to a stream of these documents, visualizing, processing, or saving them. This section provides an outline of documents themselves, aiming to give a sense of the structure and familiarity with useful components.

That makes a lot of sense.

danielballan · 2019-07-02T13:42:40Z

docs/source/event-descriptors.rst

+Event Descriptors
+=================
+
+In the section on :doc:`documents`, we gave an overview of the four kinds of


Suggested change

In the section on :doc:`documents`, we gave an overview of the four kinds of

In the section on :doc:`data-model`, we gave an overview of the types of

danielballan · 2019-07-02T13:44:07Z

docs/source/event-descriptors.rst

+=================
+
+In the section on :doc:`documents`, we gave an overview of the four kinds of
+document. We presented an example Run Start, Event, and Run Stop, but we


This second sentence is no longer accurate. We do present an Event Descriptor example. We just down break it down and discuss it in detail.

danielballan · 2019-07-02T13:44:25Z

docs/source/event-descriptors.rst

+
+.. code-block:: python
+
+    # 'event' document (same as above, shown again for reference)


"above" is out of date --- this is own its own page

danielballan · 2019-07-02T13:50:25Z

docs/source/data-model.rst

 Overview
 ========

 The data model is composed of six types of Documents, which in Python are
 represented as dictionaries but could be represented as nested mappings (e.g.
 JSON) in any language. Each document class has a defined, but flexible, schema.

+*begin remove*


I'm surprised you marked this for removal. I figure it's good to have a very short summary of the meaning of each document somewhere. As someone coming to this fresh, do you think this is so succinct that it doesn't add much value, that we might as well go straight to the more detailed description?

I would be happy with any of the following:

Drop bluesky's version of this overview and keep event-model's.

Merge some of event-model's verbiage (e.g. "the who / why / what") into bluesky's.

It depends on how much detail we want to present this early.

I liked the similar summary of each document from documents.rst because there was a lot of detail, but I see the value of a shorter summary for each document. I will drop the bluesky version.

danielballan · 2019-07-02T13:51:27Z

docs/source/data-model.rst

+
+generates a typical 'run start' document like this:
+
+.. code-block:: python


This is older and less accurate that the version in event-model. We should keep the version in event-model. We could however reincorporate the example bluesky code if that seems helpful.

jklynch · 2019-07-02T21:15:28Z

In the "Overview" section at the top of data-model.rst there is no summary for resource or datum documents. @danielballan if you would like to write those I will put them in.

danielballan · 2019-07-03T14:34:46Z

They are there; they are just below the diagram since they aren't necessary for understanding the diagram.

Would it better to move them up?

jklynch · 2019-07-09T18:27:04Z

The first sentence in the Overview section is "The data model is composed of six types of Documents...." so I feel let down because the list has only four. I would like to move the Resource and Datum documents up.

danielballan · 2019-07-09T23:57:03Z

Sounds good.

tacaswell · 2019-07-24T01:16:43Z

"Reference" is not much better than "Pointer" in terms of clarity. It's not really a uri because it is structured rather than a string.

It is "A bundle of data you can trade with another system to get the data you really want" (which is how I understand more-than-one-reference-deep pointers in c++ 😜 ).

danielballan · 2019-07-24T01:25:58Z

I'm with @prjemian that "pointer" has the potential to confuse sophisticated programmers who, in first encountering our work, provisionally map "pointer" to "memory reference" and need more context to be sure that's not what we mean. I have some empirical evidence for this: I avoided using that term at first, then tried using it for awhile, and found that it confused people. Maybe "record that points to" in place of "pointer to"?

jklynch · 2019-07-29T20:50:06Z

I suggest opening a new issue on the topic of "pointer" vs "reference" vs something else since this PR is focused on moving some specific documentation from the bluesky repo to the event-model repo.

danielballan

Great, this is getting close. Just a couple picky comments to make sure all our statements are up to date and not misleading in any way.

docs/source/data-model.rst

danielballan · 2019-07-31T14:44:55Z

docs/source/data-model.rst

+rich metadata alongside measured data for use in later analysis. Documents are
+how we do this.
+
+A *document* is our term for a Python dictionary with a schema created


This language should be updated since the RunEngine is now not the only way to create documents. Something like:

A document is our term for a Python dictionary with a schema. The bluesky RunEngine emits documents during plan execution.

We might want to elaborate:

There are tools for composing documents directly, which can be useful for ingesting data that was not acquired with bluesky into this data model.

But that might be too much detail for this section.

I am inclined to leave out the elaboration.

I agree that is too much for this context.

Great. I lean that way as well, though I included it for consideration.

prjemian

Thanks for the chance to review. Need another view before I can approve.

docs/source/data-model.rst

event_model/schemas/event_descriptor.json

prjemian

With the one additional change (Globally unique ID of this run's 'start' document.), then I'm good.

prjemian · 2019-07-31T18:21:22Z

docs/source/data-model.rst

+rich metadata alongside measured data for use in later analysis. Documents are
+how we do this.
+
+A *document* is our term for a Python dictionary with a schema created


I agree that is too much for this context.

docs/source/data-model.rst

event_model/schemas/event_descriptor.json

prjemian

That's much better!

jklynch · 2019-08-01T14:39:54Z

Sorry @prjemian I don't know why that change wasn't showing up.

danielballan

Thanks for driving this, @jklynch!

jklynch added 4 commits June 28, 2019 21:01

initial commit

7efaacf

add event-descriptors

d604d41

rename to fit in with event-model

0d0d04e

modified references to bluesky documentation so the docs can build

2606604

jklynch added the enhancement label Jun 29, 2019

jklynch self-assigned this Jun 29, 2019

first attempt at merging bluesky/docs/source/documents.rst

ecf8296

jklynch added 2 commits July 2, 2019 09:42

remove after merging with data-model.rst

7dd7239

remove after merging with data-model.rst

b8254b4

danielballan reviewed Jul 2, 2019

View reviewed changes

jklynch added 2 commits July 2, 2019 17:02

make edits based on Dan's review

90cb7a3

make edits based on Dan's review

14d10ca

jklynch added 3 commits July 23, 2019 16:34

move Resource document and Datum document up

d6b3a30

respelled 'independent'

9e850e6

no change

0c09d43

jklynch marked this pull request as ready for review July 23, 2019 20:44

danielballan mentioned this pull request Jul 30, 2019

Revisit language in docs describing Datum/Resource as "pointer" #100

Open

jklynch requested review from prjemian and danielballan July 31, 2019 14:01

danielballan reviewed Jul 31, 2019

View reviewed changes

prjemian requested changes Jul 31, 2019

View reviewed changes

edits proposed by reviewers

69cd1fc

edits proposed by reviewers

91a5537

prjemian requested changes Jul 31, 2019

View reviewed changes

jklynch requested review from prjemian and removed request for prjemian August 1, 2019 12:50

prjemian approved these changes Aug 1, 2019

View reviewed changes

jklynch requested a review from danielballan August 1, 2019 14:40

danielballan approved these changes Aug 1, 2019

View reviewed changes

danielballan merged commit f33e54a into bluesky:master Aug 1, 2019

prjemian mentioned this pull request Aug 1, 2019

Use consistent terms: run_start -- there is no such document #102

Open

	In the section on :doc:`documents`, we gave an overview of the four kinds of
	In the section on :doc:`data-model`, we gave an overview of the types of


		.. code-block:: python

		# 'event' document (same as above, shown again for reference)


		generates a typical 'run start' document like this:

		.. code-block:: python

Doc issue 78 #94

Doc issue 78 #94

Conversation

jklynch commented Jun 29, 2019

Description

Motivation and Context

How Has This Been Tested?

jklynch commented Jul 1, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jklynch commented Jul 2, 2019

danielballan commented Jul 3, 2019

jklynch commented Jul 9, 2019

danielballan commented Jul 9, 2019

tacaswell commented Jul 24, 2019

danielballan commented Jul 24, 2019

jklynch commented Jul 29, 2019 • edited Loading

danielballan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

prjemian left a comment

Choose a reason for hiding this comment

prjemian left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

prjemian left a comment

Choose a reason for hiding this comment

jklynch commented Aug 1, 2019

danielballan left a comment

Choose a reason for hiding this comment

jklynch commented Jul 29, 2019 •

edited

Loading