Uploading videos via Calibre-Web fails - 500 Internal Server Error [if filename contains square brackets]

[EMG70]

Short Notice from the maintainer

After 6 years of more or less intensive programming on Calibre-Web, I need a break.
The last few months, maintaining Calibre-Web has felt more like work than a hobby. I felt pressured and teased by people to solve "their" problems and merge PRs for "their" Calibre-Web.
I have turned off all notifications from Github/Discord and will now concentrate undisturbed on the development of “my” Calibre-Web over the next few weeks/months.
I will look into the issues and maybe also the PRs from time to time, but don't expect a quick response from me.

Please also have a look at our Contributing Guidelines

Describe the bug/problem

A clear and concise description of what the bug is. If you are asking for support, please check our Wiki if your question is already answered there.

Uploading a 24MB video failed,it returned 500 Internal Server Error.
Please see #196
To Reproduce

Steps to reproduce the behavior:

1. Go to 'https://www.youtube.com/watch?v=1xhlg9WkPQg'
2. Click on 'Upload'
3. Scroll down to '....'
4. See error
   

Logfile

Add content of calibre-web.log file or the relevant error, try to reproduce your problem with "debug" log-level to get more output.
[2024-06-28 15:11:52,277] DEBUG {cps.config_sql:407} _ConfigSQL updating storage
[2024-06-28 15:11:52,318] INFO {cps:172} Starting Calibre Web...
[2024-06-28 15:11:52,329] DEBUG {cps.services:26} Cannot import goodreads, showing authors-metadata will not work: No module named 'goodreads'
[2024-06-28 15:11:52,332] DEBUG {cps.services:34} Cannot import simpleldap, logging in with ldap will not work: No module named 'flask_simpleldap'
[2024-06-28 15:11:52,394] DEBUG {cps.services:49} Cannot import gmail, sending books via Gmail Oauth2 Verification will not work: No module named 'google_auth_oauthlib'
[2024-06-28 15:11:52,430] WARN {py.warnings:112} /usr/local/calibre-web-py3/lib/python3.12/site-packages/flask_limiter/extension.py:337: UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend.
warnings.warn(

[2024-06-28 15:11:52,691] DEBUG {cps.gdriveutils:80} Cannot import pydrive, httplib2, using gdrive will not work: No module named 'pydrive'
[2024-06-28 15:11:53,016] INFO {apscheduler.scheduler:181} Scheduler started
[2024-06-28 15:11:53,017] INFO {apscheduler.scheduler:895} Added job "delete temp" to job store "default"
[2024-06-28 15:11:53,019] INFO {apscheduler.scheduler:895} Added job "end scheduled task" to job store "default"
[2024-06-28 15:11:53,020] INFO {apscheduler.scheduler:895} Added job "immediately delete temp" to job store "default"
[2024-06-28 15:11:53,046] INFO {apscheduler.executors.default:123} Running job "immediately delete temp (trigger: date[2024-06-28 15:11:53 BST], next run at: 2024-06-28 15:11:53 BST)" (scheduled at 2024-06-28 15:11:53.019999+01:00)
[2024-06-28 15:11:53,047] INFO {apscheduler.scheduler:641} Removed job f01526ac441841a3ad30ba5ede6a56f3
[2024-06-28 15:11:53,059] DEBUG {cps.services.worker:91} Add Task for user: System - Delete Temp Folder
[2024-06-28 15:11:53,059] INFO {apscheduler.executors.default:144} Job "immediately delete temp (trigger: date[2024-06-28 15:11:53 BST], next run at: 2024-06-28 15:11:53 BST)" executed successfully
[2024-06-28 15:11:53,299] WARN {py.warnings:112} /usr/local/calibre-web-py3/cps/uploader.py:338: SyntaxWarning: invalid escape sequence ','
'-vf', 'fps=1,thumbnail,select=gt(scene,0.1),scale=-1:720', # apply filters to avoid black frames and scale

[2024-06-28 15:11:53,333] DEBUG {cps.comic:46} Cannot import comicapi, extracting comic metadata will not work: No module named 'comicapi'
[2024-06-28 15:11:53,335] DEBUG {cps.comic:53} Cannot import rarfile, extracting cover files from rar files will not work: No module named 'rarfile'
[2024-06-28 15:11:53,336] DEBUG {cps.comic:59} Cannot import py7zr, extracting cover files from CB7 files will not work: No module named 'py7zr'
[2024-06-28 15:11:53,515] DEBUG {cps.admin:82} Cannot import Flask-Dance, login with Oauth will not work: No module named 'flask_dance'
[2024-06-28 15:11:53,525] DEBUG {cps.gdrive:44} Cannot import googleapiclient, using GDrive will not work: No module named 'googleapiclient'
[2024-06-28 15:11:53,698] DEBUG {cps.search_metadata:62} Import error for metadata source: douban - No module named 'html2text'
[2024-06-28 15:11:53,726] DEBUG {cps.search_metadata:62} Import error for metadata source: scholar - No module named 'scholarly'
[2024-06-28 15:11:53,727] WARN {py.warnings:112} /usr/local/calibre-web-py3/cps/metadata_provider/lubimyczytac.py:140: SyntaxWarning: invalid escape sequence '?'
characters_to_remove = "?()/"

[2024-06-28 15:11:53,749] DEBUG {cps.search_metadata:62} Import error for metadata source: lubimyczytac - No module named 'html2text'
[2024-06-28 15:11:53,752] DEBUG {cps.search_metadata:62} Import error for metadata source: amazon - No module named 'bs4'
[2024-06-28 15:11:53,905] INFO {cps.server:269} Starting Tornado server on :8083
[2024-06-28 15:13:55,640] DEBUG {cps.web:1434} You are now logged in as: 'Admin'
[2024-06-28 15:13:55,654] DEBUG {cps.ub:82} Login and store session : 325f15e7a1c6d44ea6bc1ac28f970d6e2071c35b5a3d20293ced922c5ee82376fd9c0a53b926819be9cb978b718dcdf8c7a9b990f6731aa25de71acac4f0a718
[2024-06-28 15:13:55,655] DEBUG {cps.ub:84} Found stored session: 325f15e7a1c6d44ea6bc1ac28f970d6e2071c35b5a3d20293ced922c5ee82376fd9c0a53b926819be9cb978b718dcdf8c7a9b990f6731aa25de71acac4f0a718
[2024-06-28 15:14:14,357] DEBUG {cps.uploader:388} Temporary file: /tmp/calibre_web/37c4b0afdc8894ed8554387355402562
[2024-06-28 15:14:14,388] WARN {cps.uploader:306} Cannot find the xklb database, using default metadata
[2024-06-28 15:14:14,388] ERROR {cps:1414} Exception on /upload [POST]
Traceback (most recent call last):
File "/usr/local/calibre-web-py3/lib/python3.12/site-packages/flask/app.py", line 2190, in wsgi_app
response = self.full_dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/calibre-web-py3/lib/python3.12/site-packages/flask/app.py", line 1486, in full_dispatch_request
rv = self.handle_user_exception(e)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/calibre-web-py3/lib/python3.12/site-packages/flask/app.py", line 1484, in full_dispatch_request
rv = self.dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/calibre-web-py3/lib/python3.12/site-packages/flask/app.py", line 1469, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/calibre-web-py3/cps/usermanagement.py", line 34, in decorated_view
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/calibre-web-py3/cps/editbooks.py", line 60, in inner
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/usr/local/calibre-web-py3/cps/editbooks.py", line 251, in upload
meta, error = file_handling_on_upload(requested_file)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/calibre-web-py3/cps/editbooks.py", line 899, in file_handling_on_upload
meta = uploader.upload(requested_file, config.config_rarfile_location)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/calibre-web-py3/cps/uploader.py", line 390, in upload
return process(tmp_file_path, filename_root, file_extension, rar_excecutable)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/calibre-web-py3/cps/uploader.py", line 101, in process
if not meta.title.strip():
^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'title'
[2024-06-28 15:33:19,004] DEBUG {cps.uploader:388} Temporary file: /tmp/calibre_web/37c4b0afdc8894ed8554387355402562
[2024-06-28 15:33:19,076] WARN {cps.uploader:306} Cannot find the xklb database, using default metadata
[2024-06-28 15:33:19,077] ERROR {cps:1414} Exception on /upload [POST]
Traceback (most recent call last):
File "/usr/local/calibre-web-py3/lib/python3.12/site-packages/flask/app.py", line 2190, in wsgi_app
response = self.full_dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/calibre-web-py3/lib/python3.12/site-packages/flask/app.py", line 1486, in full_dispatch_request
rv = self.handle_user_exception(e)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/calibre-web-py3/lib/python3.12/site-packages/flask/app.py", line 1484, in full_dispatch_request
rv = self.dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/calibre-web-py3/lib/python3.12/site-packages/flask/app.py", line 1469, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/calibre-web-py3/cps/usermanagement.py", line 34, in decorated_view
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/calibre-web-py3/cps/editbooks.py", line 60, in inner
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/usr/local/calibre-web-py3/cps/editbooks.py", line 251, in upload
meta, error = file_handling_on_upload(requested_file)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/calibre-web-py3/cps/editbooks.py", line 899, in file_handling_on_upload
meta = uploader.upload(requested_file, config.config_rarfile_location)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/calibre-web-py3/cps/uploader.py", line 390, in upload
return process(tmp_file_path, filename_root, file_extension, rar_excecutable)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/calibre-web-py3/cps/uploader.py", line 101, in process
if not meta.title.strip():
^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'title'
[2024-06-28 15:35:11,771] DEBUG {cps.updater:331} Stable version: {'version': '0.6.22b'}

Expected behavior

A clear and concise description of what you expected to happen.Uploaded video is expected to successfully upload

Screenshots

If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

• OS: [e.g. Windows 10/Raspberry Pi OS]
• Python version: [e.g. python2.7]
• Calibre-Web version: [e.g. 0.6.8 or 087c4c5 (git rev-parse --short HEAD)]:
• Docker container: [None/LinuxServer]:
• Special Hardware: [e.g. Rasperry Pi Zero]
• Browser: [e.g. Chrome 83.0.4103.97, Safari 13.3.7, Firefox 68.0.1 ESR]
  

Additional context
Add any other context about the problem here. [e.g. access via reverse proxy, database background sync, special database location]

[deldesir]

@EMG70, were you trying to upload a video directly from within the YouTube menu?

[deldesir]

I just reproduce the issue @EMG70 uploading directly a video using the upload button. I am working on it.

[EMG70]

I just reproduce the issue @EMG70 uploading directly a video using the upload button. I am working on it.

Correct,I tried to upload directly using upload button .

[deldesir]

@EMG70, your upload failed because the video you tried to upload has brackets in its filename...

[holta]

upload failed because the video you tried to upload has brackets in its filename

@deldesir which brackets are currently not allowed?

(How hard is this to fix?!)

[EMG70]

Thanks @deldesir and @holta .I have removed square brackets on the video name and managed to upload the video successfully.I will try various brackets tomorrow daytime and upadate you.

[EMG70]

name of video prior to editing - GCSE Maths： Introduction to the Playlist [1xhlg9WkPQg].mp4

[deldesir]

upload failed because the video you tried to upload has brackets in its filename

@deldesir which brackets are currently not allowed?

(How hard is this to fix?!)

Square brackets were causing the issue, they are not illegal though. This is now fixed with #208

[EMG70]

Correct,I have also just confirmed this.

[EMG70]

[deldesir]

Thanks for confiming @EMG70. Please try now with the video (or another one) when you have some time. This video must have square brackets in its name.

[EMG70]

@EMG70 please try now with the video (or another one) when you have some time. This video must have square brackets in its name.

I guess I will have to do steps for testing a PR#203 right or install new VM if committed or updated on LRN2?

[deldesir]

You mean PR #208, not #203. No need to install a new VM. You follow the usual instructions to test the PR.

[EMG70]

You mean PR #208, not #203. No need to install a new VM. You follow the usual instructions to test a PR.

Sorry meant #208.I will try it.

[EMG70]

IIAB-DIAGNOSTICS - https://dpaste.com/9F6TFGRCA
Video upload with square brackets GCSE Maths： Introduction to the Playlist [1xhlg9WkPQg].mp4 now working after running PR#208 script.

[deldesir]

Thanks @EMG70. Happy this works now.

[holta]

PR #208 is now merged:

Thanks @EMG70 and @deldesir!