Implement file management #2022

jonatanklosko · 2023-06-29T22:50:41Z

First steps towards #1604. There is quite a lot of changes already, so I'm opening the PR early (or perhaps we just break it down into multiple PRs).

Overview

This adds a new sidebar with notebook files, which can either be stored (and managed) alongside the notebook in files/ directory as attachments or point to some global file or URL as links. Information about those is stored in the .livemd file. When importing a notebook from URL we know that we need to download all of its attachments.

We decided to unify this with images, so now inserting an image automatically adds an attachment and the urls are ![](files/image.jpeg) (rather than images/). We keep resolving images/ for backward compatibility for now.

UI

Expand

Sidebar

Adding from file

Adding from URL

Adding other unlisted files from files/

Deleting a file

Copying link to notebook files

Next steps

Primary

support multi-part download for large files
reading files using Kino API
links to files need to be explicitly allowed (as secrets) and we need to sign what they are
make sure files work with apps
look for image/* image urls on .livemd import and add deprecation warning

Additions

adding file via upload
global drag and drop
option to open/download a file directly from the sidebar
indicator for long file operations

github-actions · 2023-06-29T22:54:46Z

Uffizzi Preview deployment-30117 was deleted.

jonatanklosko · 2023-06-29T22:57:17Z

lib/livebook/file_system/s3.ex

+    hash = :crypto.hash(:sha256, bucket_url) |> Base.url_encode64(padding: false)
+    id = "s3-#{hash}"


Currently we are using random ids generated when the file system is added, which is fine for settings (local). However, to persist a file link in a notebook, we need a deterministic file system id, so if multiple Livebook users use the same S3 bucket, the file link is exported/imported as expected.

I did the hash so that the id is fixed length and this way it also has a predictable character set. Also, even though buckets generally need authorisation, not listing the URL explicitly sounds like an advantage to me.

I considered a second field/function like :public_id, but having both random :id and the :public_id is confusing, especially that both should be unique anyway. Also, having a deterministic :id is what we do for hubs too, so it's better for consistency. To avoid breaking existing settings I added a migration with more details.

Looks good to me. I also wonder if the ID should be the bucket itself but this at least avoids us from leaking bucket information.

Yeah, I would still like base64 encode the bucket URL to avoid weird characters in the ID, but not leaking sounds preferable anyway.

jonatanklosko · 2023-06-29T23:03:33Z

lib/livebook_web/live/session_live.ex

+            options: [
+              %{
+                name: "delete_cells",
+                label: "Delete all cells in this section",
+                default: first_section?,
+                disabled: first_section?
+              }
+            ]


When deleting a section we ask the user "Delete all cells in this section" and to do that we used a whole separate URL and modal. Now that we needed a similar option when deleting a file, I added this to the generic confirm we have, and now that it's all in Elixir it was actually very little code :D

josevalim · 2023-06-30T05:49:05Z

This looks very exciting! Here are some visual nitpicks (since I can't yet do a code review):

Move the icon folder in the sidebar before the deployments or the secrets one (don't separate deployments and runtime) (we are running out of sidebar space and we will at least have one other sidebar icon in the future but that's a separate discussion :D)
I still think the names (create a link / copy to notebook files) are not clear. First we definitely need some brief explanation somewhere: "Files are either links to existing resources or hard copies into the notebook files folder". Another idea is to break the listing by type, keeping all links and all hard copies separate. I also think we can have slightly checkbox names depending on the page:

For files, we say: "Store only file location" | "Copy file contents to the notebook files/ folder"
For URLs, we say: "Store only URL location" | "Download URL contents to the notebook files/ folder"

Finally, I am not sure if we should use the "link" icon for file links. It is typically associated with URLs/web. I would expect something closer to the icons below (but maybe i am just old :D):

jonatanklosko · 2023-06-30T10:52:21Z

Updates

zachallaun · 2023-07-01T12:08:03Z

I have some unasked-for UI/description suggestions from the perspective of a user (please feel free to disregard!)

Without some kind of formatting, usage of “files/ directory“ in a sentence is a bit confusing and looks sort of like a typo. I would just drop the trailing slash and use “files directory”
Files pane tooltip suggestion:

Manage files used by the notebook. Files are either links to existing resources or hard copies stored in a notebook files directory.
When adding a file:
- Single checkbox with a help tooltip instead of radio buttons
- For a local file: “Copy”, with tooltip “Store a copy in the notebook files directory”
- For a URL: “Download”, with tooltip “Download once and store a copy in the notebook files directory”

josevalim · 2023-07-01T12:27:50Z

I would stick with radio. The issue with checkboxes is that it doesn’t tell you what happens when it is unchecked. Radios make that clear.

agreed on “files/“, we should tag it as code (my preference) or remove the slash. :)

zachallaun · 2023-07-01T13:08:23Z

I would stick with radio. The issue with checkboxes is that it doesn’t tell you what happens when it is unchecked. Radios make that clear.

True! This may just be me, but I'd still simplify the labels pretty dramatically to something like "Link to file / Copy file" and "Link to URL / Download file" because I think that folks already have good intuition about what those things mean and too much description makes people question that intuition instead of clarifying it.

lib/livebook/migration.ex

lib/livebook_web/controllers/session_controller.ex

josevalim · 2023-07-02T09:22:41Z

lib/livebook_web/live/session_live/insert_image_component.ex

+  end
+
+  @impl true
+  def handle_event("yo", %{}, socket) do


josevalim

LGTM! Just very minor nitpicks. Something else we should add to the TODO-list is the option to open/download a file directly from the sidebar? For URLs it is easy but for files it can be a bit tricky.

josevalim · 2023-07-02T09:47:07Z

I have also added this issue, which is related to future work: #2027

Just so we don't forget. :)

lib/livebook_web/controllers/session_controller.ex

Co-authored-by: José Valim <jose.valim@dashbit.co>

lib/livebook/migration.ex

jonatanklosko · 2023-07-03T15:44:24Z

Something else we should add to the TODO-list is the option to open/download a file directly from the sidebar

Sounds good, in the future we may also want to have an zip export to download notebook with all the files.

…images

jonatanklosko · 2023-07-05T17:09:20Z

FTR automatic conversion script that will be useful for people with large notebook collections: convert_notebook_images.exs.

josevalim

We are to branch and merge :)

Implement file management

7db5998

jonatanklosko commented Jun 29, 2023

View reviewed changes

Up

b4a5ffb

josevalim reviewed Jul 2, 2023

View reviewed changes

lib/livebook/migration.ex Outdated Show resolved Hide resolved

josevalim reviewed Jul 2, 2023

View reviewed changes

lib/livebook_web/controllers/session_controller.ex Show resolved Hide resolved

josevalim reviewed Jul 2, 2023

View reviewed changes

jonatanklosko commented Jul 3, 2023

View reviewed changes

lib/livebook_web/controllers/session_controller.ex Show resolved Hide resolved

Update lib/livebook/migration.ex

42eb505

Co-authored-by: José Valim <jose.valim@dashbit.co>

jonatanklosko commented Jul 3, 2023

View reviewed changes

lib/livebook/migration.ex Show resolved Hide resolved

jonatanklosko added 3 commits July 3, 2023 17:39

Apply suggestions from code review

388f07f

Merge branch 'main' into jk-files

fbc2596

Up

8640d39

jonatanklosko added 5 commits July 3, 2023 17:51

Wording

25bafdc

Merge branch 'main' into jk-files

e2c9450

Add deprecation warning when importing a notebook referencing legacy …

d7121d0

…images

Wording

96f905e

Merge branch 'main' into jk-files

dd6d5c3

josevalim approved these changes Jul 5, 2023

View reviewed changes

jonatanklosko merged commit ede80fc into main Jul 5, 2023
7 checks passed

jonatanklosko deleted the jk-files branch July 5, 2023 18:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement file management #2022

Implement file management #2022

jonatanklosko commented Jun 29, 2023 •

edited

github-actions bot commented Jun 29, 2023 •

edited

jonatanklosko Jun 29, 2023 •

edited

josevalim Jul 2, 2023

jonatanklosko Jul 3, 2023

jonatanklosko Jun 29, 2023

josevalim commented Jun 30, 2023

jonatanklosko commented Jun 30, 2023 •

edited

zachallaun commented Jul 1, 2023 •

edited

josevalim commented Jul 1, 2023

zachallaun commented Jul 1, 2023

josevalim Jul 2, 2023

josevalim left a comment

josevalim commented Jul 2, 2023

jonatanklosko commented Jul 3, 2023

jonatanklosko commented Jul 5, 2023

josevalim left a comment

		hash = :crypto.hash(:sha256, bucket_url) \|> Base.url_encode64(padding: false)
		id = "s3-#{hash}"

Implement file management #2022

Implement file management #2022

Conversation

jonatanklosko commented Jun 29, 2023 • edited

Overview

UI

Next steps

github-actions bot commented Jun 29, 2023 • edited

jonatanklosko Jun 29, 2023 • edited

Choose a reason for hiding this comment

josevalim Jul 2, 2023

Choose a reason for hiding this comment

jonatanklosko Jul 3, 2023

Choose a reason for hiding this comment

jonatanklosko Jun 29, 2023

Choose a reason for hiding this comment

josevalim commented Jun 30, 2023

jonatanklosko commented Jun 30, 2023 • edited

zachallaun commented Jul 1, 2023 • edited

josevalim commented Jul 1, 2023

zachallaun commented Jul 1, 2023

josevalim Jul 2, 2023

Choose a reason for hiding this comment

josevalim left a comment

Choose a reason for hiding this comment

josevalim commented Jul 2, 2023

jonatanklosko commented Jul 3, 2023

jonatanklosko commented Jul 5, 2023

josevalim left a comment

Choose a reason for hiding this comment

jonatanklosko commented Jun 29, 2023 •

edited

github-actions bot commented Jun 29, 2023 •

edited

jonatanklosko Jun 29, 2023 •

edited

jonatanklosko commented Jun 30, 2023 •

edited

zachallaun commented Jul 1, 2023 •

edited