v0.5.8

Project Updates

• This packages deno in the container to take advantage of the latest major improvements from the amazing yt-dlp team. For additional context and reasoning, see here.
• This adds full metadata embedding as indexed into the media file. You can update your existing media files in bulk with an action task, docs.
  • This is recommended to run to act as an additional backup, but it is quite slow…
  • In the future that embedded metadata can be used to recreate the index.

Added

• Added automatic frontend refresh on backend restart for better cache busting, by @bbilly1
• Added full metadata embedding into video file, by @bbilly1, docs
• Added Deno runtime to the container for yt-dlp compatibility, by @bbilly1

Changed

• Changed watched filter, split to separate config values for playlists and channels, docs, by @bbilly1
• Changed similar videos, disable inline player, due to inconsistent behaviour, #1044, by @bbilly1

Fixed

• Fixed channel page size overwriting from global values, #1047, by @bbilly1
• Fixed adding to queue when page size is disabled with global overwrites, #1048, by @bbilly1
• Fixed playlist parsing, skip errors during download post processing, #1032 by @bbilly1
• Fixed upload date parsing and indexing problems, #1051 by @bbilly1
• Fixed error handling for incorrect formatted items when adding to queue, #1051 by @calexander3

v0.5.7

Project Updates

• This ships newest yt-dlp, shipping fixes for blocking issues.

Added

• Added membership integration, by @bbilly1, docs

Fixed

• Fixed frontend caching issue, by @MerlinScheurer
• Fixed error message parsing for invalid cron tabs, by @jnfbatista

Also see

• Release notes for v0.5.6
• Details and perks of the membership platform https://members.tubearchivist.com

v0.5.6

Project Updates

• In the wake of the SABR shenanigans from YT, this release ships various fixes and handling options to deal with that.
  • You might want to check if there are any low resolution videos in your archive, typically indicated by an unexpected 360p video height.
  • Use the newly introduced filter, multiselect and re-download options.
  • Ideally check your archive soon so you can redownload videos as needed, as long as they are still available on YT...
• Shoutout to the amazing yt-dlp team in providing fixes and workarounds with breakneck speeds.

Added

• Added redownload option for videos, by @bbilly1, docs
• Added multi select UI elements, by @bbilly1, docs
  • Currently implementing redownload
  • To be extended to additional actions
• Added fetching all sponsorblock segment types, by @Atinoda
• Added watched state filter null option, by @bbilly1
• Added video type filter to archive views, by @bbilly1
• Added video filter by height, by @bbilly1

Changed

• Changed ta_config index, remove indexing and field parsing to avoid type change errors, by @bbilly1
• Changed yt-dlp player used for potoken requests to recommended mweb, by @bbilly1
• Changed filter UI elements, now consolidated under filter icon, by @bbilly1
• Show subscribed only are now two different values between channel and playlist pages, by @bbilly1

Fixed

• Fixed delete videos option when deleting playlist, #1019, by @bbilly1
• Fixed add to the ignore list after auto delete, #1023 by @bbilly1
• Fixed consolidating and unify not found messaging, by @calexander3
• Fixed indexing issue for sponsorblock, define field types, by @bbilly1
• Fixed playlist incomplete index, now slicing in queue instead, #942, by @bbilly1

v0.5.5

Project Updates

• This brings various improvements when it comes to download queue management
• In our quest to reduce requests to YT, TA now internally tracks what tabs a given channel has, that’s shorts, streams and videos. Then a subscription rescan only scans the actually available tabs. That defaults to all tabs at first. The actual pages are extracted during the channel metadata refresh task.
  • To trigger the scan of all your subscribed channels you can run this command inside the TA docker container:
    python manage.py ta_index_channel_tabs.
  • That’s not strictly required as that will eventually get picked up with the refresh task, but helps to take advantage of these improvements immediately.

Added

• Added option for auto update yt-dlp at startup, by @calexander3 , docs
• Added filter by shorts, streams, regular videos in download queue, by @bbilly1, docs
• Added bulk actions in download queue, by @bbilly1, docs
• Added bulk error clearing from download queue, by @bbilly1
• Added fast add to download queue, aka bulk extract, by @bbilly1, docs
• Added additional error handling for adding to queue, by @bbilly1 #924
• Added search download queue by video title, by @bbilly1, docs
• Added playlist reverse sort order, by @bbilly1, #171, docs
• Added timestamp seeking in comments and description by @lorisleitner and @bbilly1, #677
• Added test button for new Apprise notification URLs, by @calexander3
• Added user permission mappings for LDAP, by @joshrivers, #591, docs
• Added error handling for playlist importing with multiple channels, by @bbilly1, #1009

Changed

• Changed build steps for more efficient caching, by @calexander3
• Changed removed channel manual import from json file, by @bbilly1, #1004
• Changed watch progress storing threshold, ignore below 10s, by @bbilly1, #1009
• Changed thumbnail check to also validate pending download URLs, by @bbilly1
• Changed playlist page size to a separate configurable value, by @bbilly1, docs
• Changed adding to queue per channel while scanning subscriptions, by @bbilly1
• Changed skipping scanning none existing channel tabs, by @bbilly1

Fixed

• Fixed not found loading state, by @calexander3
• Fixed unneeded referrer header for external links, by @MerlinScheurer
• Fixed unknown codec in table view, by @MerlinScheurer
• Fixed undefined stream handling, by @MerlinScheurer, #997
• Fixed language code for subscription extraction, by @oblivioncth
• Fixed duplicate notification box in channel playlist page, by @bbilly1
• Fixed mobile pagination layout wrap, by @bbilly1

v0.5.4

Changed

• bump yt-dlp

Also see the release notes for v0.5.3.

v0.5.3

Project Updates

• This triggers an index rebuild at startup for your videos to handle the new timestamp format for the date_published field. As always, this should be mostly automatic, if not see the troubleshooting guide here.
• This puts a bandaid on our scheduler problems where the scheduler doesn't schedule new jobs anymore, documented in #967. A fix would have to come from upstream, or require significant refactoring in this project to move to a different library.
• Due to the broken scheduler, It’s possible that you won’t get the convenient update notification in the footer.

Added

• Added theme handling without user context, by @MerlinScheurer
• Added "ignored" tab link on channel about page, by @Jurrer and @bbilly1
• Added "p" shortcut to play/pause toggle a video, by @bbilly1
• Added more graceful ES connection checking, validate shards, by @bbilly1
• Added additional subtitle handling options, #873 by @puigru, docs
• Added video table view layout, by @MerlinScheurer docs

Changed

• Changed "download now" when adding to queue to handle existing queued videos, #934 by @krufab
• Changed yt-dlp handling, reduce check_format calls to reduce traffic to YT, @billy1
• Changed date_published indexing to use full timestamp for better sorting, #902 by @bbilly1
• Changed authentication backend logic for forward auth and LDAP, use multiple, #591 by @joshrivers docs, docs

Fixed

• Fixed schedule input fields input UX, by @MerlinScheurer
• Fixed missing channel indexed in download queue for new installations, #932 by @bbilly1
• Fixed video player 5 seconds skip, #898 by @MerlinScheurer
• Fixed playlist video index sorting, #889 by @bbilly1
• Fixed (kind of) scheduler beat, with auto restart, #967 by @billy1
• Fixed issue for missing AppConfig and default fallback, by @skilletskills3
• Fixed stop icon handling for notification box, by @billy1

v0.5.2

Added

• Added channel name fallback to counter yt-dlp extraction errors, by @billy1

Changed

• Changed notification polling to 1sec interval to avoid some browsers blocking, #912 by @MerlinScheurer

Fixed

• Fixed URL param handling when creating embedded player, #907 by @masonjames
• Fixed newline parsing in text forms, #908 by @MerlinScheurer
• Fixed channel art fallback, #911 by @MerlinScheurer
• Fixed custom CSS injection, #884 by @MerlinScheurer, docs
• Fixed video grid type in playlist, #916 by @bbilly1
• Fixed autodelete logic for toggled watched state, #919 by @Jurrer and @bbilly1
• Fixed autodelete logic where global values can overwrite channel specific, #673 by @bbilly1
• Fixed playlist length determination, #942 by @billy1
• Fixed watched date updating logic, #926 by @Jurrer
• Fixed channel subs null value extraction, #803 by @bbilly1
• Fixed connection check to ES at startup to handle 401 errors, by @bbilly1

v0.5.1

Project Updates

• It is recommended to install v0.5.0 first before applying this update. Make sure to read the release notes there carefully and apply all manual steps.
• This is the first point release with the new React frontend, fixing and improving various things.
• This also ships the latest yt-dlp release, fixing various extraction errors. Thanks to the yt-dlp team for the awesome work!

Added

• Added “having issues” message on login failure, by @MerlinScheurer
• Added filesize unit user conf, by @MerlinScheurer and @bbilly1, #886
• Added 404 handlers for page and detail page lookup, by @MerlinScheurer and @bbilly1, #890
• Added additional sleep statements for channel playlist scan

Changed

• Health check endpoint is now at /api/health/, as updated in the compose file.

Fixed

• Fixed missing health check endpoint, #887
• Fixed missing delete and ignore button on video, #891
• Fixed DISABLE_STATIC_AUTH environment variable apply, by @trick77, #897
• Fixed appsettings lookup for none admin user, by @smantic
• Fixed channel playlist pagination, #894
• Fixed redis connection error message
• Fixed backend messages notification list type
• Fixed playlist sort, by @MerlinScheurer

v0.5.0

Project Updates, Breaking changes

• There are several breaking changes, read this carefully before updating.
  • If you accidentally updated without reading this, you can revert the image to bbilly1/tubearchivist:v0.4.13
• This ship’s the new React based frontend. Big shout out to @MerlinScheurer for taking on the bulk of the work.
• Additionally this ships a major refactor of the backend code organization.
• Also shoutout to @kralverde for helping in the backend refactor.
• This is the first iteration there might be bugs.
• The compose file is also updated, see all changes here.

Migration Guide

Local db.sqlite3

Due to the backend refactor, there were changes introduced that made persiting your user accounts and schedules unfeasible. Fear not, there is a convenient migration script for export and import.

1. On TA v0.4.13 backup your configuration by executing python manage.py ta_config_backup, e.g. docker compose exec -it tubearchivist python manage.py ta_config_backup this will create a migration file at /cache/backup/migration.json.

2. Double check that file, you should see your user(s), API token and your schedules.

3. Then stop all containers

4. Delete the db file from /cache/db.sqlite3

5. Pull the new TA image and let the initial setup complete, wait until you can reach the login page

   • You might have to set REDIS_CON here already, see bellow.
   • EDIT: If you received the update message in the TA interface, that will be in the old Redis format. You'll see an error like WRONGTYPE Operation against a key holding the wrong kind of value when TA is trying to access that. You can clear the faulty key and your migration should continue. See here.

6. Then restore the backup by executing python manage.py ta_config_restore, e.g. docker compose exec -it tubearchivist python manage.py ta_config_restore. This will restore your configurations from backup.

7. Login with your usual username/password and double check your API key and schedule config.

8. Delete /cache/backup/migration.json to avoid confusion.

Migrating the db.sqlite3 configuration is not strictly necessary. You can also just delete the file, and let it recreate at startup. You'll have to reconfigure:

• Any changes you made on your user like name/password
• The API key will have changed, you'll need to update that in e.g. the browser extension
• All schedules will be reset to default, you'll have to reconfigure them through the interface.

Redis

The configuration to connect TA with redis has changed. There is now a single environment variable called REDIS_CON to tell TA where to reach redis. If you are using the defaults, set this to redis://archivist-redis:6379. This allows for more flexibility to connect to a wide range of Redis configurations.

Additionally this project no longer depends on RedisJSON, but just on plain Redis. There is a migration step that runs at first start, you'll see ✓ migrated appconfig to ES confirming the migration. If there is nothing to migrate you'll see no config values to migrate meaning it's save to switch:

1. Stop all containers
2. From the Redis volume delete the dump.rdb file
3. Change the image from redis/redis-stack-server to just default redis.
   Note:

• you don't have to change the redis image, you could use the stack image or any compatible alternative, but you have to reset the dump.rdb file.
• this will not migrate your cookie. If you have set that, you'll have to import that again.
• this will not migrate any videos in "Continue watching", these positions will be lost.

TA_HOST protocol

If you are accessing TA behind a SSL reverse proxy, specifying the protocol is now required for the TA_HOST environment variable, e.g. https://. For the sake of consistency, also specify the protocol if you access TA without SSL, e.g. http://.
And if you are using a port in the url to access TA, you can try to add the port to the TA_HOST.
Example: TA_HOST=http://tubearchivist.local:8000

Backend port overwrite

If you previously used TA_UWSGI_PORT to modify the backend port, use the better named variable: TA_BACKEND_PORT.

Cast and Static Auth

Previously the Cast intecration was enabled with the env var ENABLE_CAST. You can now configure that in the integrations section on the config page.

There is an additional environment variable called DISABLE_STATIC_AUTH, that disables authentication on static files, required for Cast to work.

Appsettings

This is the last step for moving the redis config to ES. At startup the appsettings will get migrated from Redis to ES. That should be seamless, but depending on what values you might have set, this can create data types conflicts. At first startup, You'll see a message like:

• document_parsing_exception and failed to parse field [...] of type [...] in document with id 'appsettings'
• If you encounter that, you'll need to reset the appsettings index. From within the ES container run:

curl -XDELETE -u elastic:$ELASTIC_PASSWORD "localhost:9200/ta_config?pretty"

Then restart TA. A new blank config index will get created. You'll have to enter your config values again from the settings page.

Added

• Added additional sleep statements, by @ Styloy
• Added PO Token for yt-dlp
• Added user config toggle to show/hide help text

Changed

• Backend is now served with uvicorn, a slim and convenient asgi capable web server.
• Redis connection is now configured with the REDIS_CON environment variable for better flexibility.
• Sleep interval is not automatically randomized to +/-50% from the value set.
• There are additional sleep statements set to avoid hitting rate limits.
• The App settings page got a bit rewrite, config fields are now handled individually and not in a form.
• Similar to the channel config overwrites.
• This no longer depends on the redisJSON part of redis-server-stack but on just default redis. Simplifying things and making things less error prone for updates upstream.
• All incoming and outgoing API data and parameters are now serialized and validated.

Fixed

• Fixed live URL parsing, by @ FunkeCoder23, #805
• Fixed failing channel metadata extracting with faulty fallback implementation, #795

Dev setup

• The application can now easily be run outside of the container for development. See CONTRIBUTING.md for more details.
• Linting is now done with pre-commit for better reproducible results over various systems and CI/CD.

Docs

• All environment variables are now documented on a dedicated page link. As these apply for all installation instructions, we can avoid duplication.
• The API docs are now generated with Swagger, they are accessible on your TA instance directly at /api/docs/.
• Adding the swagger docs publically on the docs site, is pending...

API Changes

Only applicable if you made any API integrations. This is a list of changes to API endpoints.

On a general note:

• All data, queries and return statements are now serialized
• The swagger docs are accessible directly on your TA instance
• The return format of some endpoints have changed:
  • List views with pagination return a "data" top level key with a list of objects. They also return a "paginate" top level key with the pagination object.
  • List views that do not implement pagination, return the content directly in an array without a top level "data" key.
  • Detail views return the object directly without a "data" key.
  • List views no longer also return the appconfig object, use the dedicated endpoint instead.

Video endpoints:

from	to	comment
/api/video/<video_id>/progress/		removed the GET method, part of video

Task endpoints:

from	to	comment
/api/task-name/	/api/task/by-name/	get all task results
/api/task-name/<task-name>/	/api/task/by-name/<task-name>/	get task results by name
/api/task-id/<task-id>/	/api/task/by-id/<task-id>/	get single task by ID
/api/schedule/	/api/task/schedule/<task-name>/	task by name
/api/schedule/notification/	/api/task/notification/	handle apprise notifications

Settings endpoints:

from	to	comment
/api/snapshot/	/api/appsettings/snapshot/	get all ES snapshots
/api/snapshot/<snapshot-id>/	/api/appsettings/snapshot/<snapshot-id>/	single snapshot
/api/backup/	/api/appsettings/backup/	all backup files
/api/backup/<filename>/	/api/appsettings/backup/<filename>/	single backup file
/api/cookie/	/api/appsettings/cookie/	interact with cookie
/api/token/	/api/appsettings/token/	interact with API token

User endpoints:
| from | to | commen...

v0.4.13

Project Updates

• This is (probably) the last release on the old Django frontend. Next release will (hopefully) be with the new React frontend versioned v0.5.0
• There will be breaking changes in v0.5.0, be sure to read the release notes carefully before updating.

Added

• This release adds a config backup script needed for the upcoming migration to the v0.5.0 release

Fixed

• Ships latest yt-dlp resolving some 403 errors.