Notebook normalization for version control

[akhmerov]

For a while we are using nbconvert with ClearOutputPreprocessor to make notebooks git-friendly. However I have observed that there are several cell and notebook metadata fields that are left intact by it. Looking at the latest nbformat, it appears that the following cell metadata doesn't belong to VC:

• collapsed
• autoscroll
• output metadata isolated

I cannot quite find the standard kernelspec json entries, but I remember that at the very least the minor language version was causing troubles sometimes.

Now to my question: what is the best way to strip the above metadata and is there anything available short of writing an own preprocessor?

[takluyver]

nbdime can hide metadata changes, which may be some help. We also try to leave fields like those out of the metadata when they would have their default value, to minimise clutter in version control.

[akhmerov]

Yes, we use nbdime in our workflow, but I didn't realize that it can be used as a git filter; can it?

Would it be reasonable to remove output-related metadata when the corresponding output is removed?

[takluyver]

Possibly not as a filter, looks like it integrates with the diff and merge operations themselves:
http://nbdime.readthedocs.io/en/latest/vcs.html

Yes, I think metadata related to outputs should probably be cleared when the outputs are.

[akhmerov]

Without having a filter there's extra VCS noise generated.

Yes, I think metadata related to outputs should probably be cleared when the outputs are.

Where shall I open the issue? Definitely nbconvert, but also the notebook UI doesn't clean the metadata on clearing output.

[takluyver]

I'd go for it on the notebook repo.

[rsvp]

In the meantime, is there a way to clear output cells with just images?
For example, lines containing "image/png": -- without losing trust state.

A major issue in committing notebooks in repositories is, of course, the size
bulk introduced by images.

[takluyver]

I don't think there's anything ready-made to do that, but it should be pretty easy (~10 lines in Python) to load a notebook, filter out image outputs and write the result out.

[rsvp]

@takluyver I suppose the filter could be a sed one-liner, but
the notebook will probably lose its trusted state, so which
non-interactive utilities did you have in mind?

[takluyver]

I wouldn't try to sed JSON. That's not going to be remotely reliable. i was thinking Python code, using the nbformat library.

[akhmerov]

closing as answered

[minrk]

nbstripout provides git filters for notebooks for doing this kind of thing. If it doesn't cover these metadata keys, they can be added pretty easily. It may be appropriate to bring nbstripout's functionality into nbdime eventually if we want to have a single package providing "notebooks in version control" toolkit, but for now nbdime is only a renderer and merge tool.

[rsvp]

@minrk Thanks for your great suggestion. Does nbstripout
need to sign out using nbformat.sign.NotebookNotary()
to leave the modified notebook in trusted state ??

Re: stripping only images: kynan/nbstripout#58