Document workarounds for file upload size limit

[cosmoscalibur]

Is there a size limit property in FileUpload widget or a restriction for big files? Using a FileUpload with a file of 6MB, I can access to data and value fields, but when I choose a file of 10MB counter changes but data and value fields are empty.

• python=3.6.7
• notebook=6.0.0
• ipywidgets=7.5.0

Using conda-forge channel (Anaconda) for installation.

[jasongrout]

In Tornado, websocket messages have a limit of 10MB, IIRC. That may be what you are hitting.

[jasongrout]

Is there an error in the js or server side?

[cosmoscalibur]

In Tornado, websocket messages have a limit of 10MB, IIRC. That may be what you are hitting.

Is there a workaround? We need files about 20MB. When upload files web console shows:

Connection lost, reconnecting in 1 seconds. default.js:144     _onWSClose default.js:144

[jasongrout]

See jupyter/notebook#3468 for more background

[cosmoscalibur]

For future reference:

Example to limit size in 100 MB.
Create config file

jupyter notebook --generate-config

Edit (uncomment) the following line in jupyter_notebook_config.py (generated file) and change the value (desired size in MB * 1024 * 1024):

c.NotebookApp.tornado_settings = {"websocket_max_message_size": 100 * 1024 * 1024}

Launch notebook with --config parameter pointing to config file:

jupyter notebook --config="jupyter_notebook_config.py" your_notebook.ipynb

[jasongrout]

Thanks for following up with a workaround!

[jasongrout]

Do you think this is a common enough issue that we should note it in the docs?

[cosmoscalibur]

Yes, because with the popularity of Jupyter notebooks and now, the support of tools as voila to create web apps from notebooks, we can think in support upload of big files for analytics. For example, an image easily is bigger than 10 MB and a short video without codec applied also is bigger than 10 MB.
Also, for clarity, this is a limitation in the functionality of the widgets, and this is not mentioned.

[jasongrout]

Great. To be clear, this is a limitation of Tornado which impacts widgets. Would you mind writing up some text we could put in the docs around the file upload widget? Something that would have helped you a few days ago to understand the situation and work around the tornado limitation?

[jasongrout]

Or we could adapt what you wrote above, of course. Where would you have expected to find this note? In the widget list by the file upload widget? https://ipywidgets.readthedocs.io/en/latest/examples/Widget%20List.html#File-Upload

[cosmoscalibur]

I think that FileUpload is clear because is the common scenario where easily increase the size of messages but I don't know if other widget could be affected (HTML rendering with load of images and video, or image widget?). And, of course, you can adapt the text of my comment.

[jasongrout]

Thanks. I'm reopening this issue then, as an issue for adding documentation about the workaround above to the widget list, just by the file upload widget.

[jaladh-singhal]

Yes I'm getting same problem when uploading HDF files (needed for analysis), which are really big in size!

Launch notebook with --config parameter pointing to config file:
jupyter notebook --config="jupyter_notebook_config.py" your_notebook.ipynb

@cosmoscalibur @jasongrout Actually I am creating a Voila app accessible via mybinder.org so is there a way to ensure that each notebook that user opens, is by default launched with this config parameter?

[jasongrout]

@cosmoscalibur @jasongrout Actually I am creating a Voila app accessible via mybinder.org so is there a way to ensure that each notebook that user opens, is by default launched with this config parameter?

That is a great question for the voila or mybinder repos.

[jaladh-singhal]

That is a great question for the voila or mybinder repos.

@jasongrout So do you know how to find answer? What are the relevant people involved with it who can solve it?

[jasongrout]

I would either ask on the Jupyter discourse (probably better) or ask in an issue on the binderhub repo: https://github.com/jupyterhub/binderhub

[cosmoscalibur]

Yes I'm getting same problem when uploading HDF files (needed for analysis), which are really big in size!

Launch notebook with --config parameter pointing to config file:
jupyter notebook --config="jupyter_notebook_config.py" your_notebook.ipynb

@cosmoscalibur @jasongrout Actually I am creating a Voila app accessible via mybinder.org so is there a way to ensure that each notebook that user opens, is by default launched with this config parameter?

In that case @jaladh-singhal , you need to use a setup.py (use data_files field), postBuild file or dockerfile to specify the copy of the voila.py or jupyter_notebook_config.py in etc/jupyter .

[jaladh-singhal]

I would either ask on the Jupyter discourse (probably better) or ask in an issue on the binderhub repo: https://github.com/jupyterhub/binderhub

@jasongrout Thanks alot. Please let me know if you find a standard solution, other than this wonderful workaround we now know.

[jaladh-singhal]

In that case @jaladh-singhal , you need to use a setup.py (use data_files field), postBuild file or dockerfile to specify the copy of the voila.py or jupyter_notebook_config.py in etc/jupyter .

@cosmoscalibur Thanks I'll try it out and let you know if it works or not.
Just to be clear, where I should copy this jupyter_notebook_config.py in etc/jupyter,
under its root or inside any subdirectory? My directory tree of etc/jupyter is as follows:

├── jupyter_notebook_config.d
│   └── voila.json
├── jupyter_server_config.d
│   └── voila.json
└── nbconfig
    ├── common.json
    ├── edit.json
    ├── notebook.d
    │   ├── bqplot.json
    │   ├── voila.json
    │   └── widgetsnbextension.json
    ├── notebook.json
    ├── terminal.json
    └── tree.json

[cosmoscalibur]

@jaladh-singhal etc/jupyter/voila.py . Using voila, you need to change c.NotebookApp.tornado_settings for voila.

[eexwhyzee]

re: binderhub, if you want to have this config setting for all jupyter notebook servers launched off of your binder deployment (instead of having each individual repo create a jupyter notebook config using postBuild), you can add it using extraConfig in your config.yaml:

hub:
    extraConfig:
     myConfig: |
      tornado_settings_values = {
        "websocket_max_message_size": 104857600
        }
      c.KubeSpawner.args = [
        f'--NotebookApp.tornado_settings={tornado_settings_values}'
        ]

The trick is that you have to pass the tornado settings dictionary in a f-string, otherwise you'll run into an error as KubeSpawner will pass the arg as a string entirely (tornado_settings requires a dictionary)

[Kladderadatasch]

Doesn't work for me so far, but I'm also getting no errors.
I'm running Jupyter from a conda environment, would that change anything?
Is there any other way to change the file size limitation from tornado?

EDIT: Having a different conda environment / using the base Kernel makes no difference to me.

[Kladderadatasch]

I can upload files up to about 150 Kb.
Even using the torando modification in the configuration file via

"websocket_max_message_size": 100 * 1024 * 1024

Has someone any ideas on that?

[danjmp]

Just generating the file jupyter_notebook_config.py with the modification mentioned above and leaving it inside the folder .jupyter does not work for me.

"websocket_max_message_size": 100 * 1024 * 1024

What is working for me is to copy the config file to the same folder as the notebook I working on. Then I launch the notebook using the config file.

jupyter notebook --config="jupyter_notebook_config.py" my_notebook.ipynb

I hope it helps

Cheers

[mrinalraghupathi]

Is it (technically) possible to modify the javascript widget in some way to chunk the files and send it over the connection? For example as multiple 5MB messages instead of all the files in one go. I am guessing the issue occurs when the front end view tries to set the value on the model and it's all in one go?

ipywidgets/packages/controls/src/widget_upload.ts

Line 98 in 6be18d9

value: files,

[jasongrout]

Is it (technically) possible to modify the javascript widget in some way to chunk the files and send it over the connection? For example as multiple 5MB messages instead of all the files in one go. I am guessing the issue occurs when the front end view tries to set the value on the model and it's all in one go?

Yes and no. It's of course technically possible to split up a file into multiple parts and create multiple messages to stay under the 10mb limit. However, the ipywidgets protocol currently assumes each message is a self-contained state change, so we'd need to do something nontrivial changing the ipywidgets protocol to allow accumulating the results of multiple messages.

[Alexboiboi]

Just generating the file jupyter_notebook_config.py with the modification mentioned above and leaving it inside the folder .jupyter does not work for me.

"websocket_max_message_size": 100 * 1024 * 1024

What is working for me is to copy the config file to the same folder as the notebook I working on. Then I launch the notebook using the config file.

jupyter notebook --config="jupyter_notebook_config.py" my_notebook.ipynb

I hope it helps

Cheers

Hi @danjmp, I tried to use your trick with voilà but sadly the widget still only accepts 10MB.

Is there any other workaround to make this work within a voilà app?

@jasongrout any thoughts ?

[Alexboiboi]

Hi @jasongrout,

what about the "chunks" approach used by @mariobuikhuizen in ipyvuetify in
https://github.com/mariobuikhuizen/ipyvuetify/blob/master/ipyvuetify/extra/file_input.py

would there be a way to implement it directly in ipywidgets?

also open issue :
widgetti/ipyvuetify#50

[Jeannotisintheplace]

I would either ask on the Jupyter discourse (probably better) or ask in an issue on the binderhub repo: https://github.com/jupyterhub/binderhub

@jasongrout Thanks alot. Please let me know if you find a standard solution, other than this wonderful workaround we now know.

Hi there and @jaladh-singhal
Were you able to solve the max upload size issue , specially in voila?

[jaladh-singhal]

I would either ask on the Jupyter discourse (probably better) or ask in an issue on the binderhub repo: https://github.com/jupyterhub/binderhub

@jasongrout Thanks alot. Please let me know if you find a standard solution, other than this wonderful workaround we now know.

Hi there and @jaladh-singhal
Were you able to solve the max upload size issue , specially in voila?

@Jeannotisintheplace sorry I couldn't try because requirements of the project changed.

[menchyshyn]

Hi there. I can see that many people still experience the same problem. I've already read all the issue reports, stackoverflow answers and workaround solution, but I want to ask anyway: wouldn't it be better if instead of the FileUpload widget it was FilePicker returning the path to a file picked?

[jgunstone]

Hi there. I can see that many people still experience the same problem. I've already read all the issue reports, stackoverflow answers and workaround solution, but I want to ask anyway: wouldn't it be better if instead of the FileUpload widget it was FilePicker returning the path to a file picked?

this is a different (and equally valid) use-case -

when using jupyter locally on your personal machine the approach you describe might make more sense, for this you can use the ipyfilechooser

when you are interacting with remote, server hosted jupyter instance (e.g. binder, or some kind of centrally managed JupyterHub) you typically need to upload data to the server to be able to use it

[maartenbreddels]

see https://stackoverflow.com/a/75768572/5397207 for an alternative/replacement to ipyfilechooser which we use a lot to browse/pick files at the server side.

The FileUpload widget in ipywidgets is constrained to the websocket upload limit (which is a security constraint for the server). In ipyvuetify we support arbitrary filesizes and random access by sending smaller chunks over the websocket as mentioned above (see widgetti/ipyvuetify#50 for usage) and we also that approach in solara for the FileDrop component.

Maybe someone wants to port that approach over to ipywidgets.

[cgodoyle]

I'm using Voila, and a voila.json in the same folder as my notebook worked for me.

The file cointains:
{
"Voila": {
"tornado_settings": {
"websocket_max_message_size": 209715200
}
}
}

But in the end, I decided to go for the ipyvuetify.extra.FileInput solution here mentioned in the previous post, and it works perfectly in Voila as well.