Uploading of notebook and data does not work

[Chiil]

I am trying to upload one of my notebooks and the data to the pilot hub, but there seems to be no file transfer.

[choldgraf]

could you try https://2i2c.org/pilot/use.html#include-content-in-your-hub ?

[choldgraf]

also you should have an upload button here:

[Chiil]

I used the upload button, I see that the transfer has started but after that I do not see the bar filling as usual:

[choldgraf]

Hmm weird, it just worked for me. Try an nbgitpuller link and we can debug this later?

[Chiil]

The nbgitpuller works great. I have a notebook now, and the notebook can load or install the required packages. To test my full notebook, I need the upload function to work, because I cannot put our data files on GitHub.

[choldgraf]

Makes sense. I guess you could try restarting your server and see if it works after doing so. Is the dataset particular large or anything?

[Chiil]

It doesn’t help. None of the uploads work, independent of file size. I can select the file, press upload, and then it stalls.

[choldgraf]

That's bizarre, I will try to look into it though am on baby duty right now!

[choldgraf]

I just tried starting a server as your user and the upload worked fine as well...so I wonder if this could be some kind of oversees usage issue? Either way, perhaps we can find a time to debug together tomorrow morning, or perhaps @yuvipanda has ideas for what's up

another thing you could try is to use jupyterlab (just replace /tree with /lab in the URL). You should be able to upload folders/files there by dragging and dropping the files into the UI.

[Chiil]

The lab gives me the same error.

There seems to be a file size limit (~ 1 MB) in the upload function that is not there for the nbgitpuller I have made a really tiny notebook and that one I can upload.

[choldgraf]

ahh yeah I bet you're right that the file size is a bottleneck here. I just tried uploading a larger file and it didn't work as well 👍 . It sounds like a 413 error suggests this too: https://confluence.atlassian.com/jirakb/http-error-413-request-entity-too-large-failure-when-attaching-files-693900009.html#:~:text=A%20413%20error%20indicates%20that,willing%20or%20able%20to%20process.

Probably something configurable on the Jupyter server side.

In the meantime as a short-term solution to keep demo-ing, you could try to add a cell at the top of the notebook session that downloads this. e.g.:

!wget https://my-large-file

since persistent storage isn't turned on yet, you may wanna do something like this anyway to avoid having to manually upload each time.

[Chiil]

This works. My notebook runs fine and ends up with 997 MB of memory usage :). Experience has been good. Missing packages install well without trouble. I have one notebook, which is currently running, that requires ffmpeg to create movies. Would it be possible to install that? I use this, based on matplotlib:

# Call the animator. Keyword blit=True means only re-draw the parts that have changed.
anim = animation.FuncAnimation(fig1, animate, init_func=init,
                               frames=h_prime_out.shape[0], interval=40, blit=True)

HTML(anim.to_html5_video())

[yuvipanda]

The upload limit should be fixed now! I'm working on getting ffmpeg installed.

Excited to put this into production! \o/

[Chiil]

It still does not work here? I am trying to upload a 3.6 MB notebook and 50 MB of data and the same thing as before happens.

[yuvipanda]

@Chiil hmm, am investigating again now.

[yuvipanda]

@Chiil try now?

[Chiil]

Now it works! Great.

[yuvipanda]

Great! ffmpeg should also be there now.

[Chiil]

It doesn't work, maybe the script cannot find the executable.

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-10-590854a5f902> in <module>
     13                                frames=h_prime_out.shape[0], interval=40, blit=True)
     14 
---> 15 HTML(anim.to_html5_video())

/opt/conda/lib/python3.8/site-packages/matplotlib/animation.py in to_html5_video(self, embed_limit)
   1314                 # We create a writer manually so that we can get the
   1315                 # appropriate size for the tag
-> 1316                 Writer = writers[mpl.rcParams['animation.writer']]
   1317                 writer = Writer(codec='h264',
   1318                                 bitrate=mpl.rcParams['animation.bitrate'],

/opt/conda/lib/python3.8/site-packages/matplotlib/animation.py in __getitem__(self, name)
    164         if self.is_available(name):
    165             return self._registered[name]
--> 166         raise RuntimeError(f"Requested MovieWriter ({name}) not available")
    167 
    168 

RuntimeError: Requested MovieWriter (ffmpeg) not available

[choldgraf]

I wonder if you could try installing ffmpeg from conda until we get it installed on the base image:

conda install -c conda-forge ffmpeg

[yuvipanda]

I think it should be present, you might have to try starting / stopping your server. I can see it in my server

[choldgraf]

ah yes - stopping/starting made it work for me

[choldgraf]

also @Chiil - persistent storage should now be enabled, so give it a whirl if you like!

[Chiil]

It all works. My three notebooks that I tested all work out fine, and the data is persistent. Great 👍.

[choldgraf]

[choldgraf]

gonna close this one as I think it's finished 👍