Bazarr sometimes downloads subtitles too quickly before Sonarr can import external subtitle

[TChilderhose]

Describe the bug
I have a post-processing script that extracts the subtitle from video files after downloading but before importing. When Sonarr imports the files, it seems Bazarr sometimes detects it too quickly and doesn't see the subtitle file and then downloads a new file that overwrites the imported one. I'd say this happens ~20% of the time. It's not 100% consistent.

To Reproduce
Steps to reproduce the behavior:

1. Download TV show episode through Sonarr
2. Have a Post-processing script that extracts subtitles from video file
3. Let Sonarr import files (video + sub)
4. Observe that Bazarr sometimes doesnt see the imported subtitle and downloads new ones

Expected behavior
Would like to have imported Subtitles not to be overwritten.

Screenshots
N/A

Software (please complete the following information):

• Bazarr: v1.0.3-ls147 (docker)
• Radarr version 4.0.5.5981-ls134 (docker)
• Sonarr version 3.0.7.1477-ls137 (docker)
• OS: Ubuntu 20.04.4 LTS (5.4.0-104-generic x86_64)

Additional context

Naming of the files is pretty standard and set by Sonarr:

The Series Title! - S01E01 - Episode Title (1).mkv
The Series Title! - S01E01 - Episode Title (1).en.srt

Seems to be a very similar issue to #1715. I read through the ticket and got an idea from your comment here

However, I highly doubt that Radarr doing a mv on a subtitles file would take more than some milliseconds so it's very unlikely to happen.

My media lives on a ZFS pool so am wondering if that adds just enough latency that the subtitle just gets missed sometimes but purely a conjecture. Otherwise I don't know why it would only happen sometimes.

I do have a debug log file that I can send your way for a file that I seem to be able to get it to happen to at a higher frequency.
There definitely was a subtitle that was imported by Sonarr along side the video file. Main snippet though is

16/03/2022 13:20:43|DEBUG   |root                            |BAZARR syncing this specific episode from Sonarr: 53803|
16/03/2022 13:20:43|DEBUG   |peewee                          |('SELECT "t1"."path", "t1"."episode_file_id" FROM "table_episodes" AS "t1" WHERE ("t1"."sonarrEpisodeId" = ?) LIMIT ? OFFSET ?', [53803, 1, 0])|
16/03/2022 13:20:43|DEBUG   |urllib3.connectionpool          |Starting new HTTP connection (1): sonarr:8989|
16/03/2022 13:20:43|DEBUG   |urllib3.connectionpool          |http://sonarr:8989 "GET /api/system/status?apikey=(removed) HTTP/1.1" 200 None|
16/03/2022 13:20:43|DEBUG   |root                            |BAZARR got this Sonarr version from its API: 3.0.7.1477|
16/03/2022 13:20:43|DEBUG   |urllib3.connectionpool          |Starting new HTTP connection (1): sonarr:8989|
16/03/2022 13:20:43|DEBUG   |urllib3.connectionpool          |http://sonarr:8989 "GET /api/v3/episode/53803?apikey=(removed) HTTP/1.1" 200 None|
16/03/2022 13:20:43|DEBUG   |peewee.sqliteq                  |received query INSERT OR IGNORE INTO "table_episodes" ("audio_codec", "audio_language", "episode", "episode_file_id", "file_size", "format", "monitored", "path", "resolution", "scene_name", "season", "sonarrEpisodeId", "sonarrSeriesId", "title", "video_codec") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)|
16/03/2022 13:20:43|DEBUG   |peewee                          |('INSERT OR IGNORE INTO "table_episodes" ("audio_codec", "audio_language", "episode", "episode_file_id", "file_size", "format", "monitored", "path", "resolution", "scene_name", "season", "sonarrEpisodeId", "sonarrSeriesId", "title", "video_codec") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)', ['AAC', "['English']", 3, 21328, 3509620930, 'WEBRip', 'True', '/data/Video/TV/The Gilded Age/Season 1/The Gilded Age - S01E03 - Face the Music.mkv', '1080p', 'The.Gilded.Age.S01E03.Face.the.Music.1080p.HMAX.WEBRip.DD5.1.x264-NOSiViD', 1, 53803, 466, 'Face the Music', 'h264'])|
16/03/2022 13:20:43|INFO    |socketio.server                 |emitting event "data" to all [/]|
16/03/2022 13:20:43|INFO    |engineio.server                 |B2n8ZatI4dG2fyvVAAAA: Sending packet MESSAGE data 2["data",{"type":"episode","action":"update","payload":53803}]|
16/03/2022 13:20:43|DEBUG   |root                            |BAZARR inserted this episode into the database:/data/Video/TV/The Gilded Age/Season 1/The Gilded Age - S01E03 - Face the Music.mkv|
16/03/2022 13:20:43|DEBUG   |root                            |BAZARR storing subtitles for this episode: /data/Video/TV/The Gilded Age/Season 1/The Gilded Age - S01E03 - Face the Music.mkv|
16/03/2022 13:20:43|DEBUG   |root                            |BAZARR started subtitles indexing for this file: /data/Video/TV/The Gilded Age/Season 1/The Gilded Age - S01E03 - Face the Music.mkv|
16/03/2022 13:20:43|DEBUG   |root                            |BAZARR is trying to index embedded subtitles.|
16/03/2022 13:20:43|DEBUG   |peewee                          |('SELECT "t1"."episode_file_id", "t1"."file_size" FROM "table_episodes" AS "t1" WHERE ("t1"."path" = ?) LIMIT ? OFFSET ?', ['/data/Video/TV/The Gilded Age/Season 1/The Gilded Age - S01E03 - Face the Music.mkv', 1, 0])|
16/03/2022 13:20:43|DEBUG   |peewee                          |('SELECT "t1"."ffprobe_cache" FROM "table_episodes" AS "t1" WHERE ("t1"."path" = ?) LIMIT ? OFFSET ?', ['/data/Video/TV/The Gilded Age/Season 1/The Gilded Age - S01E03 - Face the Music.mkv', 1, 0])|
16/03/2022 13:20:43|DEBUG   |root                            |BAZARR returning this binary: /usr/bin/ffprobe|
16/03/2022 13:20:43|DEBUG   |peewee.sqliteq                  |received query UPDATE "table_episodes" SET "ffprobe_cache" = ? WHERE ("table_episodes"."path" = ?)|
16/03/2022 13:20:43|DEBUG   |peewee                          |('UPDATE "table_episodes" SET "ffprobe_cache" = ? WHERE ("table_episodes"."path" = ?)', [<memory at 0x7f34cedf5580>, '/data/Video/TV/The Gilded Age/Season 1/The Gilded Age - S01E03 - Face the Music.mkv'])|
16/03/2022 13:20:43|DEBUG   |peewee                          |('SELECT "t1"."code3" FROM "table_settings_languages" AS "t1" WHERE ("t1"."enabled" = ?)', [1])|
16/03/2022 13:20:43|DEBUG   |subliminal_patch.core           |external subs: scanning path /data/Video/TV/The Gilded Age/Season 1/The Gilded Age - S01E03 - Face the Music.mkv|
16/03/2022 13:20:43|DEBUG   |subliminal_patch.core           |Found subtitles {}|
16/03/2022 13:20:43|DEBUG   |subliminal_patch.core           |external subs: found {}|
16/03/2022 13:20:43|DEBUG   |peewee.sqliteq                  |received query UPDATE "table_episodes" SET "subtitles" = ? WHERE ("table_episodes"."path" = ?)|
16/03/2022 13:20:43|DEBUG   |peewee                          |('UPDATE "table_episodes" SET "subtitles" = ? WHERE ("table_episodes"."path" = ?)', ['[]', '/data/Video/TV/The Gilded Age/Season 1/The Gilded Age - S01E03 - Face the Music.mkv'])|
16/03/2022 13:20:43|DEBUG   |peewee                          |('SELECT "t1"."sonarrEpisodeId", "t1"."sonarrSeriesId" FROM "table_episodes" AS "t1" WHERE ("t1"."path" = ?)', ['/data/Video/TV/The Gilded Age/Season 1/The Gilded Age - S01E03 - Face the Music.mkv'])|
16/03/2022 13:20:43|DEBUG   |root                            |BAZARR storing those languages to DB: []|
16/03/2022 13:20:43|DEBUG   |peewee                          |('SELECT "t1"."sonarrSeriesId", "t2"."sonarrEpisodeId", "t2"."subtitles", "t1"."profileId", "t2"."audio_language" FROM "table_episodes" AS "t2" INNER JOIN "table_shows" AS "t1" ON ("t2"."sonarrSeriesId" = "t1"."sonarrSeriesId") WHERE ("t2"."sonarrEpisodeId" = ?)', [53803])|
16/03/2022 13:20:43|DEBUG   |peewee.sqliteq                  |received query UPDATE "table_episodes" SET "missing_subtitles" = ? WHERE ("table_episodes"."sonarrEpisodeId" = ?)|
16/03/2022 13:20:43|DEBUG   |peewee                          |('UPDATE "table_episodes" SET "missing_subtitles" = ? WHERE ("table_episodes"."sonarrEpisodeId" = ?)', ["['en:hi', 'en']", 53803])|
16/03/2022 13:20:43|INFO    |socketio.server                 |emitting event "data" to all [/]|
16/03/2022 13:20:43|INFO    |engineio.server                 |B2n8ZatI4dG2fyvVAAAA: Sending packet MESSAGE data 2["data",{"type":"episode","action":"update","payload":53803}]|
16/03/2022 13:20:43|INFO    |socketio.server                 |emitting event "data" to all [/]|
16/03/2022 13:20:43|INFO    |engineio.server                 |B2n8ZatI4dG2fyvVAAAA: Sending packet MESSAGE data 2["data",{"type":"badges","action":"update","payload":null}]|
16/03/2022 13:20:43|DEBUG   |root                            |BAZARR ended subtitles indexing for this file: /data/Video/TV/The Gilded Age/Season 1/The Gilded Age - S01E03 - Face the Music.mkv|
16/03/2022 13:20:43|DEBUG   |root                            |BAZARR downloading missing subtitles for this episode: /data/Video/TV/The Gilded Age/Season 1/The Gilded Age - S01E03 - Face the Music.mkv|

Is it possible to add 1 second sleep or something so see if that might be it? Or a way I can configure one? (Option, ENV variable, etc)

Let me know if you need more info. Thanks!

[halali]

Have those imported subtitles also language tag in name?

[morpheus65535]

This issue have been reported 2 times in the last week.

I'll see if I can add Defer subtitles search until next subtitles search scheduled task instead of searching immediatly for both Sonarr and Radarr. I will also make sure to provide an API endpoint to trigger a subtitles search that can be called with environment variable provided by Sonarr/Radarr custom post-processing.

@TChilderhose @anonyme123 would that help fix your similar issues?

[anonyme123]

How would this scheduled task work? Would triggering it pause all automatic subtitle search until the next manual API call?

[morpheus65535]

How would this scheduled task work? Would triggering it pause all automatic subtitle search until the next manual API call?

In fact this option would simply prevent Bazarr from searching for subtitles just after adding the episode/movie to Bazarr. It would wait for Search for wanted Series Subtitles and Search for wanted Movies Subtitles scheduled task to run before searching. Additionaly, there would be a possibility for you to call Bazarr API to trigger the search at the end of your post-processing script.

[anonyme123]

I see, yes that would totally resolve my issue!
But why make it a scheduled task, could it be a toggle in the settings?

[morpheus65535]

I see, yes that would totally resolve my issue! But why make it a scheduled task, could it be a toggle in the settings?

It's not a new scheduled task, it the that's allready there. Only the API endpoint for manual trigger of search would be new.

[anonyme123]

oh ok so basically there would be an option somewhere in the settings to not auto download subs as soon as a file is added - only relying on the regular run of 'search for wanted...' every x hours?

I thought the api endpoints already existed?:
curl -X 'PATCH' -F "radarrid=$movieID" -F "action=search-missing" "$serverIP:$bazarrPort/api/movies" -H "X-Api-Key: $bazarrAPIkey"
and
curl -X 'PATCH' -F "seriesid=$seriesID" -F "action=search-missing" "$serverIP:$bazarrPort/api/series" -H "X-Api-Key: $bazarrAPIkey"

[morpheus65535]

oh ok so basically there would be an option somewhere in the settings to not auto download subs as soon as a file is added - only relying on the regular run of 'search for wanted...' every x hours?

Exactly!

I thought the api endpoints already existed?: curl -X 'PATCH' -F "radarrid=$movieID" -F "action=search-missing" "$serverIP:$bazarrPort/api/movies" -H "X-Api-Key: $bazarrAPIkey" and curl -X 'PATCH' -F "seriesid=$seriesID" -F "action=search-missing" "$serverIP:$bazarrPort/api/series" -H "X-Api-Key: $bazarrAPIkey"

Of course I could use this one but for external integration, I prefer to define specific endpoints that won't get changed by UI breaking changes. I've done that for Plex integration.

[TChilderhose]

This issue have been reported 2 times in the last week.

I'll see if I can add Defer subtitles search until next subtitles search scheduled task instead of searching immediatly for both Sonarr and Radarr. I will also make sure to provide an API endpoint to trigger a subtitles search that can be called with environment variable provided by Sonarr/Radarr custom post-processing.

@TChilderhose @anonyme123 would that help fix your similar issues?

Yeah that would work very well for my use-case and feel like it would resolve my issue for sure.

[morpheus65535]

Ok then let me look into it in the upcoming days. I'll get back to both of you here once I have something worth trying.

[morpheus65535]

This should be fixed in upcoming beta. Check under Settings-->Sonarr/Radarr. Please report if it fix your issues.

[TChilderhose]

Just grabbed the beta and tested it with a custom script with those endpoints you added it and from my test files I had it seems to be working. Thanks for the help!

[morpheus65535]

Just grabbed the beta and tested it with a custom script with those endpoints you added it and from my test files I had it seems to be working. Thanks for the help!

Thanks for the feedback! Keep me informed!

[anonyme123]

I have to wait for the stable release in order to test it, since I've no control over the docker container provided by my host, but I'll make sure to try it once it's released.

[anonyme123]

Just tested it now that the stable release is out. It works for movies but not for episodes, it seems that newly added episodes are not automatically synced (movies are).

[morpheus65535]

Just tested it now that the stable release is out. It works for movies but not for episodes, it seems that newly added episodes are not automatically synced (movies are).

This feature that was added only prevent Bazarr from searching for subtitles on import. If your episodes aren't imported properly, it's an unrelated issue.