Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fatal error when lots of ffmpeg process doing #163

Closed
naffan2014 opened this issue Sep 3, 2021 · 19 comments
Closed

fatal error when lots of ffmpeg process doing #163

naffan2014 opened this issue Sep 3, 2021 · 19 comments

Comments

@naffan2014
Copy link

i do lots of cut work and concat work and transcode work. every work source dir and filepath is different
codes catchs lots of error. "java.io.IOException: Cannot run program "/data0/www/cache/jave/ffmpeg-amd64-2.7.3": error=26, Text file busy"
why is text file busy? how can i solve it?

@a-schild
Copy link
Owner

a-schild commented Sep 3, 2021

  • What OS are you using?
  • What JRE are you using?
  • Does it fails always, only at random 10% or at random 90% of the time?
  • Are you running multiple concurrent encodings?

Perhaps something among this line?
https://bugs.openjdk.java.net/browse/JDK-8068370

@naffan2014
Copy link
Author

  • docker : centos x86_64
  • jre : java 1.8.0_232
  • not always, it ouccrs 10% but when thread processing at a time, the num will increase to 30%
  • of course yes. i control concat mission in 3 corepollsize and 8 queueCapacity. and same as cut . so theoretically ffmpeg thread will run 6 thread at one time.

i check the articel you give me . i will change open() param FD_CLOEXEC to O_CLOEXEC. and i will observe it .

@a-schild
Copy link
Owner

a-schild commented Sep 3, 2021

Are all the threads running in the same JVM, or are they running in different JVM's?

  • If the all run in the same JVM, then there is a race condition (Missing exclusive section) between the moment we check if the executable is available in the temp location and when it's completely written to disk and flagged as executable.
  • If they run in different JVM's, then you need to put the TEMP location to different folders to prevent this problem

@naffan2014
Copy link
Author

Are all the threads running in the same JVM, or are they running in different JVM's?

  • If the all run in the same JVM, then there is a race condition (Missing exclusive section) between the moment we check if the executable is available in the temp location and when it's completely written to disk and flagged as executable.
  • If they run in different JVM's, then you need to put the TEMP location to different folders to prevent this problem

in one jvm

could u describe more about why it occurs race condition? thank you.

@naffan2014
Copy link
Author

naffan2014 commented Sep 3, 2021

this is my database persist ffmpeg working result. the column detail tells me exception reason.

<!DOCTYPE html>
<html>
<head>
<title></title>
</head>
<body>


create_time | detail | state | log_task_id
-- | -- | -- | --
2021-09-03 17:31:00 | ws.schild.jave.EncoderException\|Exit code of ffmpeg encoding run is 1 | EXCEPTION | 39
2021-09-03 17:31:01 |   | PROCESSING | 39
2021-09-03 17:31:01 | ws.schild.jave.EncoderException\|Exit code of ffmpeg encoding run is 1 | EXCEPTION | 39
2021-09-03 17:31:01 |   | PROCESSING | 39
2021-09-03 17:31:01 | ws.schild.jave.EncoderException\|Exit code of ffmpeg encoding run is 1 | EXCEPTION | 39
2021-09-03 17:31:04 | ws.schild.jave.EncoderException\|Exit code of ffmpeg encoding run is 1 | EXCEPTION | 39
2021-09-03 17:31:04 |   | PROCESSING | 39
2021-09-03 17:32:32 |   | PROCESSING | 39
2021-09-03 17:32:32 | ws.schild.jave.EncoderException\|Exit code of ffmpeg encoding run is 1 | FAILURE | 39
2021-09-03 17:32:32 | ws.schild.jave.EncoderException\|Exit code of ffmpeg encoding run is 1 | FAILURE | 39
2021-09-03 17:32:33 |   | PROCESSING | 39
2021-09-03 17:32:33 |   | PROCESSING | 39
2021-09-03 17:32:33 | ws.schild.jave.EncoderException\|Exit code of ffmpeg encoding run is 1 | FAILURE | 39
2021-09-03 17:32:33 |   | PROCESSING | 39
2021-09-03 17:32:33 | ws.schild.jave.EncoderException\|Exit code of ffmpeg encoding run is 1 | FAILURE | 39
2021-09-03 17:35:46 | Invalid data found when processing input | EXCEPTION | 45
2021-09-03 17:35:46 |   | PROCESSING | 45
2021-09-03 17:35:48 |   | PROCESSING | 45
2021-09-03 17:36:27 | Invalid data found when processing input | EXCEPTION | 45
2021-09-03 17:39:03 |   | PROCESSING | 45
2021-09-03 17:39:03 | Invalid data found when processing input | FAILURE | 45
2021-09-03 17:39:04 |   | CREATED | 46
2021-09-03 17:50:32 |   | DONE | 46
2021-09-03 17:50:32 |   | DONE | 46
2021-09-03 17:50:32 |   | PROCESSING | 46
2021-09-03 17:50:32 |   | PROCESSING | 46
2021-09-03 17:50:56 |   | CREATED | 47
2021-09-03 17:54:59 |   | CREATED | 48
2021-09-03 17:58:01 |   | PROCESSING | 46
2021-09-03 17:58:01 | java.io.IOException: Cannot run program "/data0/www/cache/jave/ffmpeg-amd64-2.7.3": error=26, Text file busy | EXCEPTION | 46
2021-09-03 17:58:01 |   | PROCESSING | 46
2021-09-03 17:58:01 | java.io.IOException: Cannot run program "/data0/www/cache/jave/ffmpeg-amd64-2.7.3": error=26, Text file busy | EXCEPTION | 46


</body>
</html>

there are three main reason in this sheet.

  • ws.schild.jave.EncoderException|Exit code of ffmpeg encoding run is 1
  • Invalid data found when processing input
  • java.io.IOException: Cannot run program "/data0/www/cache/jave/ffmpeg-amd64-2.7.3": error=26, Text file busy

@naffan2014
Copy link
Author

naffan2014 commented Sep 3, 2021

i see another issue #157. u suggest control thread through attrs.setEncodingThreads(x)。is it control ffmpeg thread number ?

@a-schild
Copy link
Owner

a-schild commented Sep 3, 2021

Could you please try again with the 3.2.0-SNAPSHOT release?

I did put in place a fix for the race condition.
5cec8b1

The ffmpeg executables are packed inside the jar files for the platforms.
When jave starts, it checks if the ffmpeg executable is found in the temp folder, if not, then it copies it from the jar file to the temp location.
If you now have two threads doing the check if the file exists in the same millisecond, then both threads will write a copy of the executable to the same location.
The first one to start the executable will then probably try to start the still "copy in progress" file of the second thread, leading to the erro 26, file busy

@a-schild
Copy link
Owner

a-schild commented Sep 3, 2021

Issue #157 tells ffmpeg to only use a certain number of threads/cores for the encoding process, but this has nothing to do with running multiple java threads which spawn the executable. (Beside using cores/cpus)

@a-schild
Copy link
Owner

a-schild commented Sep 3, 2021

The exit code 1 however is something else, most of the time some corrupt input file or some unsupported encoding/transcoding formats/settings
To see the real cause, you have to enable logging and collect the logs

@naffan2014
Copy link
Author

Could you please try again with the 3.2.0-SNAPSHOT release?

I did put in place a fix for the race condition.
5cec8b1

The ffmpeg executables are packed inside the jar files for the platforms.
When jave starts, it checks if the ffmpeg executable is found in the temp folder, if not, then it copies it from the jar file to the temp location.
If you now have two threads doing the check if the file exists in the same millisecond, then both threads will write a copy of the executable to the same location.
The first one to start the executable will then probably try to start the still "copy in progress" file of the second thread, leading to the erro 26, file busy

because some reason i cannot use above 2.7.3. i really want to use 3.x but i can't. mayby later i will make somebody to fix out maven repository bug ~

@naffan2014
Copy link
Author

i fixed the maven bug. so i can use 3.1.1 now .

let me first rebuild my code to run jave 3.1.1 . and i will send my result here or close this issue

@naffan2014
Copy link
Author

i can get 3.1.1 but i can't get 3.2.0-SNAPSHOT :(

@a-schild
Copy link
Owner

a-schild commented Sep 3, 2021

Add this in your pom.xml

<repositories>
   <repository>
     <id>oss.sonatype.org-snapshot</id>
     <url>https://oss.sonatype.org/content/repositories/snapshots</url>
     <releases>
       <enabled>false</enabled>
     </releases>
     <snapshots>
       <enabled>true</enabled>
     </snapshots>
   </repository>
 </repositories>

@naffan2014
Copy link
Author

Add this in your pom.xml

<repositories>
   <repository>
     <id>oss.sonatype.org-snapshot</id>
     <url>https://oss.sonatype.org/content/repositories/snapshots</url>
     <releases>
       <enabled>false</enabled>
     </releases>
     <snapshots>
       <enabled>true</enabled>
     </snapshots>
   </repository>
 </repositories>

sorry

Add this in your pom.xml

<repositories>
   <repository>
     <id>oss.sonatype.org-snapshot</id>
     <url>https://oss.sonatype.org/content/repositories/snapshots</url>
     <releases>
       <enabled>false</enabled>
     </releases>
     <snapshots>
       <enabled>true</enabled>
     </snapshots>
   </repository>
 </repositories>

i use gradle ,not maven~ do u know how to add these in gradle ?

@a-schild
Copy link
Owner

a-schild commented Sep 9, 2021

@naffan2014
Copy link
Author

Not sure, but probably something like this:

https://docs.gradle.org/current/userguide/declaring_repositories.html#sec:repository-content-filtering ?

thank u . i can get 3.2.0-SNAPSHOT now

@naffan2014
Copy link
Author

when i use 3.2.0-snapshot . i can;t do the concat mp4 work again. issue is #166

@a-schild
Copy link
Owner

@naffan2014 Can we close this issue and continue on #166 ?

@naffan2014
Copy link
Author

@naffan2014 Can we close this issue and continue on #166 ?

yes。i close this issue at once.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants