fileserver can't handle long file names

[ranrubin]

Hi,

Got 500 from the server while training, after looking in the logs I found out it was because of the file name.

Logs from /opt/trains/logs/fileserver.log

[2020-07-12 14:22:04,977] [7] [ERROR] [fileserver] Exception on / [POST]
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.6/site-packages/flask_cors/extension.py", line 161, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "fileserver.py", line 32, in upload
    file.save(str(target))
  File "/usr/local/lib/python3.6/site-packages/werkzeug/datastructures.py", line 3066, in save
    dst = open(dst, "wb")
OSError: [Errno 36] File name too long: '/mnt/fileserver/trains/14-trains.bd72a5a2afdhsy2aa0acc3dca21b9b5f/metrics/Evaluator CV_no_my_real_name_no_my_real_name_no_my_real_name_len_512_Jul12_14-20-42_merge__no_my_real_nameh_sz_8__no_my_real_name_sz_8_lr_1e-06_w_decay_0.0_warm_up_50_in_sz_768_hid_sz_256_word_aug_p_0.0_no_my_real_name_1__no_my_real_name/_no_my_real_name _no_my_real_name_no_my_real_namele layer__no_my_real_namelen_512_Jul12_14-20-42__no_my_real_name_batch_sz_8_test_batch_sz_8_lr_1e-06_w_d_no_my_real_name0_in_sz_768_hid__no_my_real_name_0.0_word_aug_min_1_imba_no_my_real_name__no_my_real_name_00000000.jpeg'

[jkhenning]

Hi @ranrubin,

Was the 500 response returned from the fileserver? Your initial post indicated the fileserver has crashed, but it looks to me like a 500 response was returned when Trains attempted to upload the debug image. If that's the case, there's little we can do about it but limit the filename length in the Trains SDK when uploading, as this is an OS limit.

What do you think?

[ranrubin]

Sorry, I will provide the exact response once I will manage to reproduce the issue.
Generally speaking, I would rather get a message in the UI saying that the file name is too big, rather than have to look for the issue in the logs :)

[jkhenning]

Generally speaking, I would rather get a message in the UI saying that the file name is too big, rather than have to look for the issue in the logs :)

I couldn't agree more 🙂
Since this error is encountered by Trains running as part of the experiment code (whether it is returned form the file server, or even if Trains will detect an abnormally long file name before even trying to upload it) - it's all in the realm of the experiment, and other than reporting it in the log, what you're actually describing is a way for the UI to clearly display all kinds of errors encountered by the experiment (including, but not limited to, file names being too long...) - this is a great feature request - can you maybe open a feature request issue for this? 😉

[ranrubin]

opened #54 :)