Open code in local editor

[okennedy]

Currently code editing has to happen in the notebook itself. This is cumbersome for large files and user-hostile.

Suggestion: Add a feature to open code in a local editor.

Desiderata

(in decreasing-order of value)

• The ability to use an arbitrary text editor of your choosing to edit files.
• Click-to-edit from a web user interface
• Integration with e.g., mypy/flake8/IDE code completion/etc... (e.g., by declaring 'placeholder' files in the same directory)
• Support for arbitrary websites, not just this one

Implementation Strategies

Create and monitor a temp file.

Dump the cell script to something like /tmp/[xxxxxxx].py and use open (OSX) or xdg-open (Linux) to open the script in the user's preferred editor. Monitor the script file for edits (e.g., polling or using inode watchers or something similar) and re-run the notebook whenever an edit is saved. This is simple and easy to pull off, but only works on local Vizier instances.

WebDAV

Have the server expose a WEBDAV filesystem listing available notebooks, with a file per (script) cell. The difficulty here is that right now module identifiers are versioned. That means the filename would change with every iteration. For this to work, we'd need a new "version independent" identifier in addition to the version.

The advantage is that this would work regardless of whether we're running desktop or server vizier, but doesn't give us a neat 'open' button on the UI.

FUSE

Like WebDAV, but use FUSE. This provides more reliable edit detection and client-side experience, but requires users to install fuse and a client-side app. With this requiring a client, however, an 'open' button would be possible.

[okennedy]

Possibly supplants VizierDB/web-ui#92

[okennedy]

A fourth option would be to create a script that runs locally and does the same thing as option 1. Register this script with a URL handler (OSX and Linux both support this) and have it talk to the API.

[okennedy]

I wonder if there's a Chrome/Firefox plugin that lets you edit the content of a text field in a text editor.

[okennedy]

Alternatively, we use flask to implement a really really simple webdav server. That might not actually be all that hard.

[okennedy]

I'm going to simplify the goals here a little bit and focus exclusively on getting this implemented for use in the desktop version.

[okennedy]

For this to work, we need a way to signal the UI that code is being edited/updated. That pretty much means websockets (#59 ). Deferring this feature until later.

[sakshi-mehra]

@okennedy Question: When does the file watcher/ agent stop watching the file for changes? Is it when the file is closed by the user or shall we have put some TTL or both?
Also, I was curious to find out any robust ways to check when/if a file is closed.

[okennedy]

@okennedy Question: When does the file watcher/ agent stop watching the file for changes? Is it when the file is closed by the user or shall we have put some TTL or both? Also, I was curious to find out any robust ways to check when/if a file is closed.

This is a good question. I don't have a great answer. Even if it is a temporary directory, you can't assume that it will get cleaned out with any frequency. This is a somewhat common pattern for [S]FTP editor support (e.g., SublimeText's FTP plugin, CyberDuck). I would suggest doing some market research to see what options are used, and what their tradeoffs are.

[okennedy]

A prototype of this issue was implemented by a 611 team. Thanks to Aniruddha Thaksen Parvat, Nikhil Naresh Vatwani, Paramveer Singh, @sakshi-mehra , Soma Naga Venkata Sai Anvesh Motamarri, and Sphoorthi Keshannagari.

The prototype is in 4 parts:

• UI Changes: https://github.com/param1111/web-ui
• API Changes: https://github.com/sakshi-mehra/vizier-scala
• Stub Server: https://github.com/aniruddha96/StubServer
• Vizier-Agent: https://github.com/aniruddha96/vizier-agent

The central component is an agent deployed on the client's machine (this could be integrated into the backend server for Vizier desktop) that is responsible for opening an editor and monitoring the edited file for changes. When the file contents change, the agent performs a cell update on the relevant cell. Notably, the agent uses a URL scheme to communicate with the UI (probably move this to the Websocket in the desktop version), which is nice, since the agent doesn't need to be running to work.

Supporting MyPy

One of the goals of the project was to be able to use automatic code completion, type linting, etc...

1. Code completion and linting is not possible for libraries on the server unless those libraries (with the exact same version) are also available locally. This point was addressed by downloading "stub" files (type information) from the server and making it available to mypy through the config file. Stub files are generated and hosted by a "stub server".
2. The 'vizierdb' object needs to be in-scope. This was solved by inserting a "DO NOT EDIT ME" header at the top of the file importing relevant symbols.

There are a few limitations:

1. At present all stub files for the server are downloaded. This is maybe overkill, but there's no way to easily figure out when mypy requests a stub file to download it on-demand. A few ideas:
   • WebDAV would allow introspecting requests for stub files... but this makes the agent quite messy.
   • Use some sort of type analysis to figure out which library symbols are required... but this precludes autocompletion until the cell is saved.
   • Have the agent cache stub files and share between instances... but this could get tricky with multiple python versions
2. There is a potential conflict with Jupyter Import #18 , specifically in terms of allowing cells to dynamically access artifacts without going through vizierdb["artifact"]. We could include explicit imports of the relevant artifacts in the header section, but if the set of artifacts changes upstream of the cell, the only way to fix it would be to modify the file in-situ (which risks deleting a user's edits)

Further Design Considerations

1. Tracking user sessions is hard. The agent can't really tell when the document is closed. This was solved by adding a button to the UI that allows the user to close the editor session. The agent then updates the file locally with a big warning saying "THIS FILE IS NO LONGER BEING SYNCED" and quits.
   • Alternative idea: Many editors have a "blocking" mode (e.g., subl -w). Have the user configure an editor and use that to monitor for when the file is closed.