Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make hls output write atomic. #3286

Merged
merged 1 commit into from
Aug 1, 2023
Merged

Make hls output write atomic. #3286

merged 1 commit into from
Aug 1, 2023

Conversation

toots
Copy link
Member

@toots toots commented Aug 1, 2023

Fixes: #3284

@toots toots added this pull request to the merge queue Aug 1, 2023
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Aug 1, 2023
@toots toots added this pull request to the merge queue Aug 1, 2023
Merged via the queue into main with commit 51342b3 Aug 1, 2023
23 checks passed
@toots toots deleted the hls-atomic-files branch August 1, 2023 19:08
toots added a commit that referenced this pull request Aug 1, 2023
@BusterNeece
Copy link

BusterNeece commented Aug 2, 2023

@toots Running into this error now on our Docker installations:

 Source /var/azuracast/stations/azuratest_radio/hls/live_m3u8 failed while streaming: Sys_error("Invalid cross-device link")!
2023/08/02 05:05:32 [clock.main:2] Raised by primitive operation at Hls_output.hls_output#open_out.object#close in file "src/core/outputs/hls_output.ml", line 548, characters 16-44
2023/08/02 05:05:32 [clock.main:2] Re-raised at Hls_output.hls_output#open_out.object#close in file "src/core/outputs/hls_output.ml", line 553, characters 16-52
2023/08/02 05:05:32 [clock.main:2] Called from Hls_output.hls_output#close_segment.(fun) in file "src/core/outputs/hls_output.ml", line 568, characters 13-51
2023/08/02 05:05:32 [clock.main:2] Called from Stdlib__Option.map in file "option.ml", line 24, characters 57-62
2023/08/02 05:05:32 [clock.main:2] Called from Hls_output.hls_output#close_segment in file "src/core/outputs/hls_output.ml", line 566, characters 8-924
2023/08/02 05:05:32 [clock.main:2] Called from Hls_output.hls_output#write_pipe.(fun) in file "src/core/outputs/hls_output.ml", line 945, characters 13-33
2023/08/02 05:05:32 [clock.main:2] Called from Stdlib__Option.map in file "option.ml", line 24, characters 57-62
2023/08/02 05:05:32 [clock.main:2] Called from Hls_output.hls_output#write_pipe in file "src/core/outputs/hls_output.ml", line 941, characters 8-226
2023/08/02 05:05:32 [clock.main:2] Called from Stdlib__List.iter2 in file "list.ml", line 147, characters 24-31
2023/08/02 05:05:32 [clock.main:2] Called from Output.encoded#send_frame.output_chunks.f in file "src/core/outputs/output.ml", line 238, characters 10-24
2023/08/02 05:05:32 [clock.main:2] Called from Output.output#output in file "src/core/outputs/output.ml", line 179, characters 45-70
2023/08/02 05:05:32 [clock.main:2] Called from Clock.MkClock.clock#end_tick.(fun) in file "src/core/clock.ml", line 318, characters 16-24

That folder isn't a bind-mounted volume, nor is the temp directory, so they should theoretically be in the same "filesystem" as far as Docker is concerned.

@toots
Copy link
Member Author

toots commented Aug 2, 2023

Thanks @BusterNeece. It would appear that the temporary directory is on a different file system than the hls directory.

I just pushed changes in main and rolling-release-v2.2.x to:

  1. Fallback to non-atomic file write instead of failing
  2. Add an optional temp_dir option to output.hls to make it possible to specify a temporary directory in the same file system.

Would you be kind enough to check that these two changes do work for you? The script should now run by default but in degraded, non-atomic mode and you should be able to specify a temporary directory in the same partition, restoring atomic write operations.

Commit is 49c1c27 in main and 47464a4 in rolling-release-v2.2.x

Thanks!

@BusterNeece
Copy link

@toots Thanks, will update to that. Seems the CI is red since that commit went live so we don't have a release asset for it yet.

@toots
Copy link
Member Author

toots commented Aug 2, 2023

@BusterNeece should be good now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

HLS CRASH - no EXTM3U delimiter
2 participants