Notebook improvements

[vfdev-5]

Feature motivation

I have a question about Notebooks. When I start a notebook for development in working env purposes, I see a treeview with files of the project (default workspace). But when I open a terminal in the notebook, the files seen there (at /code) are not the same as in the default workspace.
In terminal : /code
Workspace: /repos/user/project/...
Maybe it would be better to unify this ?
When I'm happy with the modifications done in jupyter I would like to download files locally with polyaxon project download. But this downloads /code and not the workspace...
Where could we also see the Outputs tab for notebook ?

[mmourafiq]

I agree, it's a bit confusing, the reason is that there are 2 use cases:

• the user has a multi-read-write volume for code, i.e. the user will be able to commit the code after finishing the notebook
• the user does not have that option, and the notebook is using code either from github or internal repo, so must be downloaded.

I will improve the notebook docker build process to remove this confusing, and maybe write about it to make people aware of the use-cases and the limitation.

Thanks for being patient with this.

[Mofef]

At the moment if I want to save a Notebook I was working on, I download it to my local project dir. So next time I can continue from there.
In general I would also prefer if persistence was ensured some other way. (e.g. In case the machine running the notebook unexpectedly dies)

[kdubovikov]

@mouradmourafiq , the exact functionality we need is already provided by NOTEBOOKS_MOUNT_CODE option. Everything is already implemented in polypod.

However, the option is missing from the UI.

I have tried to setting NOTEBOOKS:MOUNT_CODE to True using Settings->Other Options->More Options screen, but my notebooks do not start now. The error message is the following: "exist-code(1)-message(None)". Could you point to where the problem might be?

I also wonder if there is any way to get logs for failed notebook job…

[mmourafiq]

NOTEBOOKS_MOUNT_CODE was confusing for users, also it's not clear how it would work with external repos. The whole thing also requires a clear documentation. So for now, I would just say that the feature is not implemented yet.

[kdubovikov]

@mouradmourafiq , for the external repos I suppose you could just git commit && git push from the notebook. Also, JupyterLab has Git integration plugin.

[mmourafiq]

Thanks for the feedback. I agree, but putting a documentation simple and clear for now is not obvious.
there's a WIP to improve the notebook experience, which will enhance among other things, the code access/management. I hope that it will land on the v0.6.

[mmourafiq]

Closed as well in #690.

A couple of notes about service, notebooks included, in the v1:

• Users can start a notebook and attach connections, these secrets could include SSH and Git auth tokens, this would allow to pull/push more easily.
• Users can instead or in addition to mounting the ssh/token connections, ask to pre-prepare the notebook with a list of github repos, data, outputs, or list of artifacts in general.
• polyaxon ops logs can be used in similar way for jobs and services, and it will print all logs in case a notebook/service has an issue.
• Service can generate as well artifacts and outputs, and so it will be similar to the experiments outputs.
• If the notebooks are saved in the outputs directory, users can get notebook versioning and resuming automatically.