Notebook metadata proposal (title, subtitle, authors, affiliations)

[FRidh]

While creating a presentation with the Notebook and Reveal.js I noticed it would be nice to add certain metadata to the notebook that could then be accessed in the template. Therefore, I would like to make a proposal of the metadata fields regarding the documents title, subtitle and affiliations.

Metadata proposal

I propose we separate authors from affiliations and use dictionaries to access them.
An affiliation is divided into 'institute, department and section'. Often all three are used on a title page, while only the institute is shown in the header/footer.

Regarding author names I propose we use the LaTeX way of Lastname/Firstname.
However, perhaps we should also separate the authors name like with the affiliations, having separate fields for the first, middle and last name. What do you think?

I've grouped these fields together into document, because I expect there will be a lot more metadata in the future, e.g. exporter specific settings.

{
"name": "",
"celltoolbar": "Slideshow",
"signature": "sha256:b01826fe48973cffb8307654e0cb5e48b963c0e44cb0013e9d5c5a5ebc13c321",

"document": {
    "title": "Title of document/presentation",
    "subtitle": "Subtitle of document/presentation",
    "authors": {
        "author1": {
            "name": "LastName, FirstName",
            "affiliation": "org1",
            "email" : "author1@domain.org",
            },
        "author2": {
            "name": "LastName, FirstName",
            "affiliation": "org2",
            "email" : "author2@domain.de",
            },
        },
    "affiliations": {
        "org1" : {
            "institute" : "University of...",
            "department": "Department of...",
            "section"   : "Laboratory of...",
            "street"    : "Street A",
            "city"      : "City A",
            "country"   : "Country A",
            "postcode"  : "Postcode A",
            },
        "org2" : {
            "institute" : "... GmbH",
            "department": "Department of...",
            "section"   : "Section...",
            "street"    : "Street A",
            "city"      : "City A",
            "country"   : "Country A",
            "postcode"  : "Postcode A",
            },
        },
    },
}

As you can see these are quite a lot of new fields. While many won't we used that often I think it is good to "standardize" the metadata.

Implementation

During conversion from the notebook to any other format these fields have to be accessible. If I am correct resources['metadata'] should store these values.

At which point should this ResourcesDict be filled with the metadata?
I suppose an additional preprocessor should be written that adds document to resources['metadata'] and that recursively converts all dictionaries to ResourcesDict?

Accessing metadata in templates

The metadata can then be accessed in templates, e.g. the title by calling
{{resources['metadata']['document']['title']}}

[dsblank]

This would be nice! Thinking about utility of the data, it would be nice for a future nbviewer to be able to render a bibliography reference for download/cu-n-paste in a couple of formats (eg, bibtex, MLA, APA). For completeness, I'd suggest just mirroring all of bibtex's fields.

[FRidh]

I hadn't thought of that, but indeed, I think it would be straightforward to implement a function then to take this meta data and print a limited .bib file entry or vCard's of the authors.

[dsblank]

If this proposal was about a document metadata section, I would also suggest that we add: license, license_logo, and license_text (could include HTML). And that would also appear when editing or nbviewing the notebook.

[ellisonbg]

In the past we have resisted adding this much formalized metadata to the
notebook format. Not sure we are ready to jump into this. There are also a
number of other ways we have discussed adding this type of information
without putting it into metadata.

On Wed, Jul 2, 2014 at 4:01 PM, Doug Blank notifications@github.com wrote:

If this proposal was about a document metadata section, I would also
suggest that we add: license, license_logo, and license_text (could
include HTML). And that would also appear when editing or nbviewing the
notebook.

—
Reply to this email directly or view it on GitHub
#6073 (comment).

Brian E. Granger
Cal Poly State University, San Luis Obispo
@ellisonbg on Twitter and GitHub
bgranger@calpoly.edu and ellisonbg@gmail.com

[FRidh]

Do you happen to have any references to to mailings or issues here where that's discussed? I would really like to know why this was.

[bgamari]

@ellisonbg would you happen to have some references to discussion surrounding this issue? It appears to be a bit Google-resistant.

One idea might be to use the existing Markdown YAML metadata syntax (see, for instance, the implementation in Pandoc's markdown). It's a bit unclear how one would deal with cases like multiple metadata-containing cells, however.

Either way, metadata would be rather nice. The current recommended means of changing the document author leaves much to be desired.

[ellisonbg]

Unfortunately, I think that most of these discussions have happened in
person and in our G+ dev meetings over a long period of time. So I am not
sure there is a better discussion than this one.

A short summary of some of the points:

• This type of metadata is a rats nest of complexity. Many people have
  invented ways of encoding this type of information and we don't want to
  reinvent the wheel, but also I am not sure there is a good standard.
• It is not just in the metadata that we might want this information.
  Eventually you would want to display this information, both in a live
  notebook and in nbconvert (HTMl, PDF, etc.). What that UI looks like is
  non-trivial.
• Formats. The way you encode this type of data varys a lot with the
  desired output format HTML, latex, etc.
• Once we make a decision, folks will start to use it and we will have
  invented a new standard! That means we need to get it roughly right the
  first time.

We would love someone to look into these questions. But in areas where we
are creating a standard, we move very slowly and conservatively. Cheers,
Brian

On Thu, Oct 2, 2014 at 9:18 AM, Ben Gamari notifications@github.com wrote:

@ellisonbg https://github.com/ellisonbg would you happen to have some
references to discussion surrounding this issue? It appears to be a bit
Google-resistant.

One idea might be to use the existing Markdown YAML metadata syntax (see,
for instance, the implementation in Pandoc's markdown
http://johnmacfarlane.net/pandoc/README.html#extension-yaml_metadata_block).
It's a bit unclear how one would deal with cases like multiple
metadata-containing cells, however.

Either way, metadata would be rather nice. The current recommended means
of changing the document author leaves much to be desired.

—
Reply to this email directly or view it on GitHub
#6073 (comment).

Brian E. Granger
Cal Poly State University, San Luis Obispo
@ellisonbg on Twitter and GitHub
bgranger@calpoly.edu and ellisonbg@gmail.com

[bollwyvl]

I propose treating, initially, every metadata block, but eventually, each entire notebook, as JSON-LD documents. Basically, this means adding/fiating a @context to every metadata block. Then adding some namespaces of linked data. Then starting to do useful stuff with it.

JSON-LD provides a standards-based way to capture a much richer underlying structure to JSON data than JSON schema, for example. This "linked data" concept asserts that data has:

• identity, @id
• type, @type
• as defined by a @context

The example above with the multiple affiliations is trivial, not only to write, but to actually do something with, as there exist parsers in most major languages. Further, the specific pidgen

The biggest initial win would be in selecting and promoting a few, existing, productive namespaces for 99% of the cases of metadata that someone might want.

• The biggest player in this space is dublin core. Scholarly. If I had to pick one, I'd probably just go with this.
• The biggest challenger is schema.org, which is what powers, for example, Google Now, and that thing that lets you do stuff to github issues from inside gmail. For an SEO/internet citizenship point of view, this is very possibly the future.
• The biggest vision is W3C PROV, which would actually contain the means to capture the complex relationship between multiple simultaneous users. and uses dublin core as a baseline.

Now, the joke is that we could claim that we are using all of them, as one can make their OWN term, then use ontology dark magic to claim the it is the same as other standards. For example, jupyter:name could be described to link to rdfs:label, dc:title and schema:name. However, i recommend against doing such a thing, aside from those places where we are really doing new things, i.e. ComputableDocument is a subclass of schema:CreativeWork. JupyterKernel is a subclass of schema:Software.

Longer term, JSON-LD provides things like internationalization, out-of-band context and other features that would contribute substantially to the overall impact of the IPython/Jupyter ecosystem.

Since the data-at-rest representation is still JSON, this introduces no real new challenges, but gives us a huge head start: we already have some great collections of concepts to pull in, rather than having to build the UI and the vocabulary at the same time.

[ellisonbg]

I think for the time being we need to be extremely conservative in not
changing the notebook format. The cost to the project and users in changing
the format is massive. That doesn't prevent others from playing around with
different things in metadata though in the meantime.

Cheers,

Brian

On Sat, Mar 21, 2015 at 10:20 PM, bollwyvl notifications@github.com wrote:

I propose treating, initially, every metadata block, but eventually, each
entire notebook, as JSON-LD http://json-ld.org documents. Basically,
this means adding/fiating a @context
http://json-ld.org/spec/ED/json-ld-syntax/20111016/#the-context to
every metadata block. Then adding some namespaces of linked data. Then
starting to do useful stuff with it.

JSON-LD provides a standards-based way to capture a much richer underlying
structure to JSON data than JSON schema, for example. This "linked data"
concept asserts that data has:

• identity, @id
• type, @type
• as defined by a @context

The example above with the multiple affiliations is trivial, not only to
write, but to actually do something with, as there exist parsers in most
major languages. Further, the specific pidgen

The biggest initial win would be in selecting and promoting a few,
existing, productive namespaces for 99% of the cases of metadata that
someone might want.

• The biggest player in this space is dublin core
  http://dublincore.org/. Scholarly. If I had to pick one, I'd
  probably just go with this.
• The biggest challenger is schema.org, which is what powers, for
  example, Google Now, and that thing that lets you do stuff to github issues
  from inside gmail. For an SEO/internet citizenship point of view, this is
  very possibly the future.
• The biggest vision is W3C PROV http://www.w3.org/TR/prov-primer/,
  which would actually contain the means to capture the complex relationship
  between multiple simultaneous users. and uses dublin core as a baseline.

Now, the joke is that we could claim that we are using all of them, as
one can make their OWN term, then use ontology dark magic to claim the it
is the same as other standards. For example, jupyter:name could be
described to link to rdfs:label, dc:title and schema:name. However, i
recommend against doing such a thing, aside from those places where we are
really doing new things, i.e. ComputableDocument is a subclass of
schema:CreativeWork. JupyterKernel is a subclass of schema:Software.

Longer term, JSON-LD provides things like internationalization,
out-of-band context and other features that would contribute substantially
to the overall impact of the IPython/Jupyter ecosystem.

Since the data-at-rest representation is still JSON, this introduces no
real new challenges, but gives us a huge head start: we already have some
great collections of concepts to pull in, rather than having to build the
UI and the vocabulary at the same time.

—
Reply to this email directly or view it on GitHub
#6073 (comment).

Brian E. Granger
Cal Poly State University, San Luis Obispo
@ellisonbg on Twitter and GitHub
bgranger@calpoly.edu and ellisonbg@gmail.com

[bollwyvl]

I think for the time being we need to be extremely conservative in not changing the notebook format.

Agreed. That's one of the big advantages of JSON-LD: we can just say that the notebook is now JSON-LD, as interpreted by a particular context, and it is. But there is still a lot of information missing... my motivation here is the seemingly simple nbviewer issues, and influenced by making it work with, for example, google drive.

A ContentManager is, in the end, is responsible for giving the notebook a URL. Heck, I like the index.ipynb suggestion. But this suggests that the filename is a thing for machines, that the human might not even control. What is the name for people? At present, google is nice enough to pull an h1 out of the content of notebooks, but an empty title is really pretty bad for humans to find it.

I'm saying that when we add "real" metadata for metadata's sake (that some robot from outside the ecosystem would actually recognize as such), JSON-LD would be the right way to capture it.

So that when we enable populating the title of notebooks on nbviewer, even if it is to fall back to the filename, that someone should be able to override it. And when they provide it, and maybe still in dialog.js, not only as hand-edited JSON, but they should be able to pick from a collection of vocabularies that provide value beyond their current browser or nbviewer.

My preferred solution for the UI/UX challenge would be, however, to move the meta linked data out of a dark modal, and put it in the main UI, much like Lyx. Being able to point to a cell and promote it to The Title, or The Abstract, and then have real formatting applied to it (.jumbotron, blockquote) and know that that would show up in google as the link title and excerpt would be very powerful.

Endgame

Longer term, as was discussed on the list, really thinking about the Notebook document from the ground up as linked data would be incredible, and allow things like:

• slide remixing
• copy/paste with attribution
• multiple human languages
• publishing of discoverable data along with notebooks
• and eventually... being able to use the ecosystem of notebooks as something computable... along the lines of the wolfram language