(1) download.py: "sqlite3.OperationalError: no such column: error" after "TypeError: 'NoneType' object is not subscriptable" (2) editbooks.py: SAWarning: Object of type <Books> not in session, add operation along 'Authors.books' won't proceed #186

holta · 2024-06-14T18:15:10Z

22 of 29 videos downloaded for @EMG70 (76% is a start, but also not that great!) when he tried to download...

"Precious Plastic Academy"
https://www.youtube.com/playlist?list=PLXpbdfI05JvRT07cgyZjwcfSYGiFgnbtM

Two main issues appear to be:

download.py: sqlite3.OperationalError: no such column: error after TypeError: 'NoneType' object is not subscriptable (appeared just once?)
editbooks.py: SAWarning: Object of type <Books> not in session, add operation along 'Authors.books' won't proceed (appears often!)

Excerpt from his iiab-diagnostics https://dpaste.com/C8SLESTWU showing both:

-rw-r--r-- 1 root root 94883 Jun 14 17:34 /var/log/calibre-web.log
                        ...ITS LAST 100 LINES FOLLOW...

[2024-06-14 17:31:52,496]  INFO {cps.tasks.download:106} Successfully sent the requested file to http://192.168.0.28/books/meta
[2024-06-14 17:31:52,509]  INFO {cps.tasks.download:129} Download task for https://www.youtube.com/watch?v=NuYj4hDjKzU completed successfully
[2024-06-14 17:31:52,509]  INFO {cps.tasks.download:41} Subprocess args: ['lb-wrapper', 'dl', 'https://www.youtube.com/watch?v=w6Hyd9c5QMw']
[2024-06-14 17:32:13,810]  INFO {cps.editbooks:385} Received metadata request: ImmutableMultiDict([('requested_file', '/library/downloads/calibre-web/Youtube/One Army/Precious Plastic - Start a business with plastic recycling (part 6.0)_35.80k_[w6Hyd9c5QMw].mp4'), ('current_user_name', 'Admin'), ('shelf_id', '4')])
[2024-06-14 17:32:13,810]  INFO {cps.editbooks:387} Requested file: /library/downloads/calibre-web/Youtube/One Army/Precious Plastic - Start a business with plastic recycling (part 6.0)_35.80k_[w6Hyd9c5QMw].mp4
[2024-06-14 17:32:13,811]  INFO {cps.editbooks:392} Processing file: <_io.BufferedReader name='/library/downloads/calibre-web/Youtube/One Army/Precious Plastic - Start a business with plastic recycling (part 6.0)_35.80k_[w6Hyd9c5QMw].mp4'>
[2024-06-14 17:32:13,811] DEBUG {cps.uploader:376} Temporary file: /tmp/calibre_web/84b88f2fea2e0d667acc017fdc57c534
[2024-06-14 17:32:13,821]  WARN {py.warnings:110} /usr/local/calibre-web-py3/cps/editbooks.py:1535: SAWarning: Object of type <Books> not in session, add operation along 'Authors.books' won't proceed (This warning originated from the Session 'autoflush' process, which was invoked automatically in response to a user-initiated operation.)
  db_element = db_session.query(db_object).filter((func.lower(db_filter).ilike(add_element))).first()

[2024-06-14 17:32:13,832] DEBUG {cps.helper:548} Moving title: /tmp/calibre_web/84b88f2fea2e0d667acc017fdc57c534 to /library/calibre-web/One Army/Precious Plastic - Start a business with plastic recycling (part 6.0) (29)/Precious Plastic - Start a business with plastic recycling (part 6.0) - One Army
[2024-06-14 17:32:13,873]  INFO {cps.tasks.download:106} Successfully sent the requested file to http://192.168.0.28/books/meta
[2024-06-14 17:32:13,887]  INFO {cps.tasks.download:129} Download task for https://www.youtube.com/watch?v=w6Hyd9c5QMw completed successfully
[2024-06-14 17:32:13,887]  INFO {cps.tasks.download:41} Subprocess args: ['lb-wrapper', 'dl', 'https://www.youtube.com/watch?v=BtrfTSSGHEo']
[2024-06-14 17:32:28,072] ERROR {cps.tasks.download:122} An error occurred during the subprocess execution: 'NoneType' object is not subscriptable
[2024-06-14 17:32:28,073] ERROR {cps.services.worker:202} no such column: error
Traceback (most recent call last):
  File "/usr/local/calibre-web-py3/cps/tasks/download.py", line 89, in run
    requested_file = conn.execute("SELECT path FROM media WHERE webpath = ? AND path NOT LIKE 'http%'", (self.media_url,)).fetchone()[0]
                     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^
TypeError: 'NoneType' object is not subscriptable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/calibre-web-py3/cps/services/worker.py", line 199, in start
    self.run(*args)
  File "/usr/local/calibre-web-py3/cps/tasks/download.py", line 123, in run
    self.message = f"{self.media_url_link} failed to download: {self.read_error_from_database()}"
                                                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/calibre-web-py3/cps/tasks/download.py", line 139, in read_error_from_database
    error = conn.execute("SELECT error FROM media WHERE webpath = ?", (self.media_url,)).fetchone()[0]
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
sqlite3.OperationalError: no such column: error

His screenshot:

The text was updated successfully, but these errors were encountered:

holta · 2024-06-14T19:11:41Z

@deldesir

SAWarning: Object of type <Books> not in session, add operation along 'Authors.books' won't proceed (appears often!)

Do the notes & "potential fix" below from @codewiz help with the (SQLAlchemy) SAWarning?

https://chatgpt.com/share/1c5737a6-e9fc-4eee-83e2-9aff755ed0a2

holta · 2024-06-16T00:01:49Z

Thanks Much @deldesir for posting your ideas here when you can!

holta · 2024-06-24T19:16:19Z

@deldesir please try to reproduce both errors this week if you have a moment?🕯️

holta · 2024-07-01T22:03:31Z

@deldesir is this (#186) still an issue?

(Or is it possibly/definitely resolved?)

Possibly related:

deldesir · 2024-07-01T22:56:08Z

I cannot reproduce this issue. I cannot spot the error messages in my tests.

holta · 2024-07-02T04:57:40Z

@EMG70 can you please re-test?

18+ days ago your downloading of playlist "Precious Plastic Academy"
https://youtube.com/playlist?list=PLXpbdfI05JvRT07cgyZjwcfSYGiFgnbtM only worked with 22 of 29 videos!
Does a fresh install of IIAB Calibre-Web now download all 29 of its videos ?

EMG70 · 2024-07-02T05:04:29Z

I will test today and update you.

EMG70 · 2024-07-02T18:17:59Z

Tested on a new VM
SUDO IIAB-DIAGNOSTICS -https://dpaste.com/BS87LW36V

Playlist https://www.youtube.com/playlist?list=PLXpbdfI05JvRT07cgyZjwcfSYGiFgnbtM 24/29 Videos successfully downloaded .

The 5 failed videos were attempted individually and all failed as well.

holta · 2024-07-02T18:52:30Z

24/29 Videos successfully downloaded

24 is better than 22 (from 3 weeks ago!)

@deldesir the ~5 errors in /var/log/calibre-web.log (same for each failed video?) would appear to be exactly as (1) download.py: "sqlite3.OperationalError: no such column: error" after "TypeError: 'NoneType' object is not subscriptable" in the subject line of this ticket.
For example starting on Line 1182 for https://youtu.be/dP1s7viFZHY pasted in below.
Please help explain what appears to be happening — Thank You.

(Whereas @EMG70 no longer experienced (2) editbooks.py: SAWarning: Object of type <Books> not in session, add operation along 'Authors.books' won't proceed in the subject line of this ticket. Did PR #193 really solve that aspect?)

[2024-07-02 19:14:48,096] ERROR {cps.tasks.download:128} An error occurred during the subprocess execution: 'NoneType' object is not subscriptable
[2024-07-02 19:14:48,096] ERROR {cps.services.worker:202} no such column: error
Traceback (most recent call last):
  File "/usr/local/calibre-web-py3/cps/tasks/download.py", line 89, in run
    requested_file = conn.execute("SELECT path FROM media WHERE webpath = ? AND path NOT LIKE 'http%'", (self.media_url,)).fetchone()[0]
                     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^
TypeError: 'NoneType' object is not subscriptable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/calibre-web-py3/cps/services/worker.py", line 199, in start
    self.run(*args)
  File "/usr/local/calibre-web-py3/cps/tasks/download.py", line 129, in run
    self.message = f"{self.media_url_link} failed to download: {self.read_error_from_database()}"
                                                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/calibre-web-py3/cps/tasks/download.py", line 145, in read_error_from_database
    error = conn.execute("SELECT error FROM media WHERE webpath = ?", (self.media_url,)).fetchone()[0]
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
sqlite3.OperationalError: no such column: error

holta · 2024-07-06T15:35:23Z

Thanks to @nzola for uncovering another example here:

Formerly live https://youtu.be/4BL65HElOPg fails to download [same "no such column: error" issue as #186 ?] #215

holta added bug Something isn't working question Further information is requested labels Jun 14, 2024

holta assigned deldesir Jun 16, 2024

holta pinned this issue Jun 24, 2024

holta mentioned this issue Jul 6, 2024

Formerly live https://youtu.be/4BL65HElOPg fails to download [same "no such column: error" issue as #186 ?] #215

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

(1) download.py: "sqlite3.OperationalError: no such column: error" after "TypeError: 'NoneType' object is not subscriptable" (2) editbooks.py: SAWarning: Object of type <Books> not in session, add operation along 'Authors.books' won't proceed #186

(1) download.py: "sqlite3.OperationalError: no such column: error" after "TypeError: 'NoneType' object is not subscriptable" (2) editbooks.py: SAWarning: Object of type <Books> not in session, add operation along 'Authors.books' won't proceed #186

holta commented Jun 14, 2024 •

edited

Loading

holta commented Jun 14, 2024

holta commented Jun 16, 2024

holta commented Jun 24, 2024

holta commented Jul 1, 2024 •

edited

Loading

deldesir commented Jul 1, 2024

holta commented Jul 2, 2024

EMG70 commented Jul 2, 2024

EMG70 commented Jul 2, 2024 •

edited

Loading

holta commented Jul 2, 2024 •

edited

Loading

holta commented Jul 6, 2024

(1) download.py: "sqlite3.OperationalError: no such column: error" after "TypeError: 'NoneType' object is not subscriptable" (2) editbooks.py: SAWarning: Object of type <Books> not in session, add operation along 'Authors.books' won't proceed #186

(1) download.py: "sqlite3.OperationalError: no such column: error" after "TypeError: 'NoneType' object is not subscriptable" (2) editbooks.py: SAWarning: Object of type <Books> not in session, add operation along 'Authors.books' won't proceed #186

Comments

holta commented Jun 14, 2024 • edited Loading

holta commented Jun 14, 2024

holta commented Jun 16, 2024

holta commented Jun 24, 2024

holta commented Jul 1, 2024 • edited Loading

deldesir commented Jul 1, 2024

holta commented Jul 2, 2024

EMG70 commented Jul 2, 2024

EMG70 commented Jul 2, 2024 • edited Loading

holta commented Jul 2, 2024 • edited Loading

holta commented Jul 6, 2024

holta commented Jun 14, 2024 •

edited

Loading

holta commented Jul 1, 2024 •

edited

Loading

EMG70 commented Jul 2, 2024 •

edited

Loading

holta commented Jul 2, 2024 •

edited

Loading