Enable users to copy both files and directories #1190

kenyaachon · 2023-01-30T05:42:57Z

Objective

Enable users to copy both files and directories
Currently, if users try copying a directory, an error will thrown backing saying "Can't Copy directories".

jupyterlab

The copy option in the right click menu, was enabled for directories.

jupter_server

The copy directory functionality has been added to both the FileContentsManager and AsyncFileContentsManager
Users can copy folders up to 100 MB,
- The actual limit for copying directories before a timeout occurs is 500MB, but without a loading indicator, if a user tries to copy a folder that is near 500MB, jupyterlab will seem like its frozen, since its not clear whether the copy operation is complete or in progress.
- if we can add a loading indicator in the future to jupyterlab, then we would be able to increase the limit of copying folders to 500 MB

How to test

Create a virtual environment with venv (you can use what ever viritual environment tool you like)

python3 -m venv jupyter-connected
source jupyter-connected/bin/activate
pip install --upgrade pip
# install the jupyter_server package into the environment
git clone https://github.com/kenyaachon/jupyter_server.git
cd jupyter_server
git checkout copy-folder-functionality
pip install -e ".[test]"
# verify the dev version (i.e 2.1.0.dev0) of the package is in the environment
pip list | grep jupyter
# install the jupyterlab in development mode with latest changes
cd ..
git clone https://github.com/kenyaachon/jupyterlab.git
cd jupyterlab
git checkout develop
pip install -e .
jlpm install
jlpm run build 
jupyter lab --dev-mode

Limitations

We were not able to find a suitable solution yet for being able to copy folders above 500MB in size without having a timeout issue
Users, cannot easily know if the folder they are copying is above 100MB before they try and copy
- We did attempt in adding the folder size to the preview info that users get when hovering over a file or folder. But this ended up making the filebrowser-extension very slow when loading the correct directory. Since it would have to recalculate the size of the folders in the given directory each time a new path was loaded.

welcome · 2023-01-30T05:42:59Z

Thanks for submitting your first pull request! You are awesome! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please make sure you followed the pull request template, as this will help us review your contribution more quickly.

You can meet the other Jovyans by joining our Discourse forum. There is also a intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

codecov · 2023-01-30T05:46:06Z

Codecov Report

Base: 80.45% // Head: 80.28% // Decreases project coverage by -0.17% ⚠️

Coverage data is based on head (395842b) compared to base (903942c).
Patch coverage: 82.90% of modified lines in pull request are covered.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1190      +/-   ##
==========================================
- Coverage   80.45%   80.28%   -0.17%     
==========================================
  Files          68       68              
  Lines        8135     8249     +114     
  Branches     1586     1602      +16     
==========================================
+ Hits         6545     6623      +78     
- Misses       1176     1202      +26     
- Partials      414      424      +10

Impacted Files	Coverage Δ
jupyter_server/services/contents/filemanager.py	`76.93% <82.75%> (+1.18%)`	⬆️
jupyter_server/services/contents/manager.py	`82.99% <100.00%> (-2.27%)`	⬇️
jupyter_server/extension/handler.py	`67.69% <0.00%> (-1.54%)`	⬇️
...ter_server/services/kernels/connection/channels.py	`61.06% <0.00%> (-0.67%)`	⬇️
jupyter_server/gateway/managers.py	`83.37% <0.00%> (-0.24%)`	⬇️
jupyter_server/serverapp.py	`79.96% <0.00%> (-0.09%)`	⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

blink1073 · 2023-01-30T14:21:01Z

Hi @kenyaachon, thank you for working on this! Where are you seeing a timeout, on the server or the browser?

kenyaachon · 2023-01-30T15:39:06Z

Hi @kenyaachon, thank you for working on this! Where are you seeing a timeout, on the server or the browser?

Hi @blink1073 I was seeing the timeout on the server. Let me reproduce a screenshot of the error.

kenyaachon · 2023-02-03T18:42:54Z

@blink1073 Here is the timeout error that occurs,

It seems with the latest version of the jupyter-server, this error no longer occurs, in the other project that I developed this code from, we are using version 1.18.0 which had that limitation. The timeout error would occur when copying folders around 500MB, but it seems now I can copy folders around >1GB. It seems it would be fine to remove the code around limiting how big of a folder can be copied, but I am not sure if that might cause problems on less powerful machines.

blink1073 · 2023-02-03T19:35:19Z

We can expose this as a setting, a new trait on ContentsManager like (max_copy_folder_size_mb), and then whoever is deploying the server can make that determination. 500MB seems like a reasonable default.

for more information, see https://pre-commit.ci

blink1073 · 2023-02-10T16:09:42Z

@kenyaachon, can you please add that new trait as part of this PR?

kenyaachon · 2023-02-11T03:34:19Z

@kenyaachon, can you please add that new trait as part of this PR?

I will make sure to add the trait as part of the review, once I am done addressing the failing tests.

blink1073 · 2023-02-11T15:37:29Z

Great, thanks! The failing tests are unrelated to this PR.

for more information, see https://pre-commit.ci

blink1073 · 2023-02-14T15:31:10Z

jupyter_server/services/contents/manager.py

@@ -121,6 +133,8 @@ def _notary_default(self):
    """,
    )

+    max_copy_folder_size_mb = Int(500, config=True, help="The max folder size that can be copied")


Hmm, perhaps this trait should be on FileManager, since it isn't implemented in the more generic ContentsManager. Anyone have any thoughts?

Yes - I completely agree - good catch.

I will make sure to make that change @blink1073.

@blink1073, when you say FileManager do you mean the FileContentsManager or the LargeFileManager?

Hi @kenyaachon - yes, FileContentsManager is the appropriate location for the trait since it's the "most base" class for the file-based contents managers. Sorry for the confusion.

…ntentsManager

blink1073

Looks good, thank you!

welcome · 2023-02-24T13:47:38Z

Congrats on your first merged pull request in this project! 🎉

Thank you for contributing, we are very proud of you! ❤️

vidartf · 2023-03-03T16:11:23Z

Most windows installs do not have du command, so the _get_dir_size method fails. Any chance that this can be fixed? Or at least that we disable the test on Windows? (or maybe even a check whether du is available as a callable, and skips the test if not?)

blink1073 · 2023-03-03T16:56:52Z

Yeah, I think checking for which("du") makes sense, PR welcome!

blink1073 added the enhancement label Jan 30, 2023

kenyaachon force-pushed the copy-folder-functionality branch from d6a8eba to 52254d4 Compare January 30, 2023 15:52

kenyaachon mentioned this pull request Jan 30, 2023

Add copy option for directories jupyterlab/jupyterlab#13886

Merged

mmbugua and others added 4 commits February 9, 2023 16:45

Enable users to copy both files and directories

f890ef4

Fix issues for windows test for the test_copy_big_dir

57200e8

[pre-commit.ci] auto fixes from pre-commit.com hooks

7fc89fb

for more information, see https://pre-commit.ci

Fix tests for macos platform

aa8b252

kenyaachon force-pushed the copy-folder-functionality branch from 7afa176 to aa8b252 Compare February 10, 2023 00:45

kenyaachon and others added 2 commits February 9, 2023 18:29

Fix linting issues

80df337

[pre-commit.ci] auto fixes from pre-commit.com hooks

d7520ed

for more information, see https://pre-commit.ci

kenyaachon force-pushed the copy-folder-functionality branch from d5c0471 to d7520ed Compare February 10, 2023 03:24

kenyaachon and others added 5 commits February 11, 2023 20:02

add max_copy_folder_size_mb trait and adjust tests

8ab6fb5

fixup! add max_copy_folder_size_mb trait and adjust tests

79a0192

fixup! add max_copy_folder_size_mb trait and adjust tests

8f0bdf3

[pre-commit.ci] auto fixes from pre-commit.com hooks

6092b31

for more information, see https://pre-commit.ci

remove old code that is commented out

e5d65a1

blink1073 reviewed Feb 14, 2023

View reviewed changes

Move the max_copy_folder_size_mb trait from ContentsManager to FileCo…

f859307

…ntentsManager

blink1073 approved these changes Feb 24, 2023

View reviewed changes

Merge branch 'main' into copy-folder-functionality

395842b

blink1073 merged commit bc828c5 into jupyter-server:main Feb 24, 2023

vidartf mentioned this pull request Mar 3, 2023

Skip dir size check if not enumerable #1227

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable users to copy both files and directories #1190

Enable users to copy both files and directories #1190

kenyaachon commented Jan 30, 2023

welcome bot commented Jan 30, 2023

codecov bot commented Jan 30, 2023 •

edited

blink1073 commented Jan 30, 2023

kenyaachon commented Jan 30, 2023 •

edited

kenyaachon commented Feb 3, 2023 •

edited

blink1073 commented Feb 3, 2023

blink1073 commented Feb 10, 2023

kenyaachon commented Feb 11, 2023

blink1073 commented Feb 11, 2023

blink1073 Feb 14, 2023

kevin-bates Feb 14, 2023

kenyaachon Feb 18, 2023

kenyaachon Feb 22, 2023

kevin-bates Feb 22, 2023

blink1073 left a comment

welcome bot commented Feb 24, 2023

vidartf commented Mar 3, 2023

blink1073 commented Mar 3, 2023

Enable users to copy both files and directories #1190

Enable users to copy both files and directories #1190

Conversation

kenyaachon commented Jan 30, 2023

Objective

jupyterlab

jupter_server

How to test

Limitations

welcome bot commented Jan 30, 2023

codecov bot commented Jan 30, 2023 • edited

Codecov Report

blink1073 commented Jan 30, 2023

kenyaachon commented Jan 30, 2023 • edited

kenyaachon commented Feb 3, 2023 • edited

blink1073 commented Feb 3, 2023

blink1073 commented Feb 10, 2023

kenyaachon commented Feb 11, 2023

blink1073 commented Feb 11, 2023

blink1073 Feb 14, 2023

Choose a reason for hiding this comment

kevin-bates Feb 14, 2023

Choose a reason for hiding this comment

kenyaachon Feb 18, 2023

Choose a reason for hiding this comment

kenyaachon Feb 22, 2023

Choose a reason for hiding this comment

kevin-bates Feb 22, 2023

Choose a reason for hiding this comment

blink1073 left a comment

Choose a reason for hiding this comment

welcome bot commented Feb 24, 2023

vidartf commented Mar 3, 2023

blink1073 commented Mar 3, 2023

codecov bot commented Jan 30, 2023 •

edited

kenyaachon commented Jan 30, 2023 •

edited

kenyaachon commented Feb 3, 2023 •

edited