fatal error when lots of ffmpeg process doing #163

naffan2014 · 2021-09-03T07:19:37Z

i do lots of cut work and concat work and transcode work. every work source dir and filepath is different
codes catchs lots of error. "java.io.IOException: Cannot run program "/data0/www/cache/jave/ffmpeg-amd64-2.7.3": error=26, Text file busy"
why is text file busy? how can i solve it?

a-schild · 2021-09-03T07:26:36Z

What OS are you using?
What JRE are you using?
Does it fails always, only at random 10% or at random 90% of the time?
Are you running multiple concurrent encodings?

Perhaps something among this line?
https://bugs.openjdk.java.net/browse/JDK-8068370

naffan2014 · 2021-09-03T08:22:41Z

docker : centos x86_64
jre : java 1.8.0_232
not always, it ouccrs 10% but when thread processing at a time, the num will increase to 30%
of course yes. i control concat mission in 3 corepollsize and 8 queueCapacity. and same as cut . so theoretically ffmpeg thread will run 6 thread at one time.

i check the articel you give me . i will change open() param FD_CLOEXEC to O_CLOEXEC. and i will observe it .

a-schild · 2021-09-03T09:48:12Z

Are all the threads running in the same JVM, or are they running in different JVM's?

If the all run in the same JVM, then there is a race condition (Missing exclusive section) between the moment we check if the executable is available in the temp location and when it's completely written to disk and flagged as executable.
If they run in different JVM's, then you need to put the TEMP location to different folders to prevent this problem

naffan2014 · 2021-09-03T09:59:08Z

Are all the threads running in the same JVM, or are they running in different JVM's?

If the all run in the same JVM, then there is a race condition (Missing exclusive section) between the moment we check if the executable is available in the temp location and when it's completely written to disk and flagged as executable.

If they run in different JVM's, then you need to put the TEMP location to different folders to prevent this problem

in one jvm

could u describe more about why it occurs race condition? thank you.

naffan2014 · 2021-09-03T10:05:16Z

this is my database persist ffmpeg working result. the column detail tells me exception reason.

<!DOCTYPE html>
<html>
<head>
<title></title>
</head>
<body>


create_time | detail | state | log_task_id
-- | -- | -- | --
2021-09-03 17:31:00 | ws.schild.jave.EncoderException\|Exit code of ffmpeg encoding run is 1 | EXCEPTION | 39
2021-09-03 17:31:01 |   | PROCESSING | 39
2021-09-03 17:31:01 | ws.schild.jave.EncoderException\|Exit code of ffmpeg encoding run is 1 | EXCEPTION | 39
2021-09-03 17:31:01 |   | PROCESSING | 39
2021-09-03 17:31:01 | ws.schild.jave.EncoderException\|Exit code of ffmpeg encoding run is 1 | EXCEPTION | 39
2021-09-03 17:31:04 | ws.schild.jave.EncoderException\|Exit code of ffmpeg encoding run is 1 | EXCEPTION | 39
2021-09-03 17:31:04 |   | PROCESSING | 39
2021-09-03 17:32:32 |   | PROCESSING | 39
2021-09-03 17:32:32 | ws.schild.jave.EncoderException\|Exit code of ffmpeg encoding run is 1 | FAILURE | 39
2021-09-03 17:32:32 | ws.schild.jave.EncoderException\|Exit code of ffmpeg encoding run is 1 | FAILURE | 39
2021-09-03 17:32:33 |   | PROCESSING | 39
2021-09-03 17:32:33 |   | PROCESSING | 39
2021-09-03 17:32:33 | ws.schild.jave.EncoderException\|Exit code of ffmpeg encoding run is 1 | FAILURE | 39
2021-09-03 17:32:33 |   | PROCESSING | 39
2021-09-03 17:32:33 | ws.schild.jave.EncoderException\|Exit code of ffmpeg encoding run is 1 | FAILURE | 39
2021-09-03 17:35:46 | Invalid data found when processing input | EXCEPTION | 45
2021-09-03 17:35:46 |   | PROCESSING | 45
2021-09-03 17:35:48 |   | PROCESSING | 45
2021-09-03 17:36:27 | Invalid data found when processing input | EXCEPTION | 45
2021-09-03 17:39:03 |   | PROCESSING | 45
2021-09-03 17:39:03 | Invalid data found when processing input | FAILURE | 45
2021-09-03 17:39:04 |   | CREATED | 46
2021-09-03 17:50:32 |   | DONE | 46
2021-09-03 17:50:32 |   | DONE | 46
2021-09-03 17:50:32 |   | PROCESSING | 46
2021-09-03 17:50:32 |   | PROCESSING | 46
2021-09-03 17:50:56 |   | CREATED | 47
2021-09-03 17:54:59 |   | CREATED | 48
2021-09-03 17:58:01 |   | PROCESSING | 46
2021-09-03 17:58:01 | java.io.IOException: Cannot run program "/data0/www/cache/jave/ffmpeg-amd64-2.7.3": error=26, Text file busy | EXCEPTION | 46
2021-09-03 17:58:01 |   | PROCESSING | 46
2021-09-03 17:58:01 | java.io.IOException: Cannot run program "/data0/www/cache/jave/ffmpeg-amd64-2.7.3": error=26, Text file busy | EXCEPTION | 46


</body>
</html>

there are three main reason in this sheet.

ws.schild.jave.EncoderException|Exit code of ffmpeg encoding run is 1
Invalid data found when processing input
java.io.IOException: Cannot run program "/data0/www/cache/jave/ffmpeg-amd64-2.7.3": error=26, Text file busy

naffan2014 · 2021-09-03T10:12:53Z

i see another issue #157. u suggest control thread through attrs.setEncodingThreads(x)。is it control ffmpeg thread number ?

a-schild · 2021-09-03T10:30:08Z

Could you please try again with the 3.2.0-SNAPSHOT release?

I did put in place a fix for the race condition.
5cec8b1

The ffmpeg executables are packed inside the jar files for the platforms.
When jave starts, it checks if the ffmpeg executable is found in the temp folder, if not, then it copies it from the jar file to the temp location.
If you now have two threads doing the check if the file exists in the same millisecond, then both threads will write a copy of the executable to the same location.
The first one to start the executable will then probably try to start the still "copy in progress" file of the second thread, leading to the erro 26, file busy

a-schild · 2021-09-03T10:31:18Z

Issue #157 tells ffmpeg to only use a certain number of threads/cores for the encoding process, but this has nothing to do with running multiple java threads which spawn the executable. (Beside using cores/cpus)

a-schild · 2021-09-03T10:32:34Z

The exit code 1 however is something else, most of the time some corrupt input file or some unsupported encoding/transcoding formats/settings
To see the real cause, you have to enable logging and collect the logs

naffan2014 · 2021-09-03T10:46:44Z

Could you please try again with the 3.2.0-SNAPSHOT release?

I did put in place a fix for the race condition.
5cec8b1

The ffmpeg executables are packed inside the jar files for the platforms.
When jave starts, it checks if the ffmpeg executable is found in the temp folder, if not, then it copies it from the jar file to the temp location.
If you now have two threads doing the check if the file exists in the same millisecond, then both threads will write a copy of the executable to the same location.
The first one to start the executable will then probably try to start the still "copy in progress" file of the second thread, leading to the erro 26, file busy

because some reason i cannot use above 2.7.3. i really want to use 3.x but i can't. mayby later i will make somebody to fix out maven repository bug ~

naffan2014 · 2021-09-03T11:41:09Z

i fixed the maven bug. so i can use 3.1.1 now .

let me first rebuild my code to run jave 3.1.1 . and i will send my result here or close this issue

naffan2014 · 2021-09-03T12:08:09Z

i can get 3.1.1 but i can't get 3.2.0-SNAPSHOT :(

a-schild · 2021-09-03T12:10:28Z

Add this in your pom.xml

<repositories>
   <repository>
     <id>oss.sonatype.org-snapshot</id>
     <url>https://oss.sonatype.org/content/repositories/snapshots</url>
     <releases>
       <enabled>false</enabled>
     </releases>
     <snapshots>
       <enabled>true</enabled>
     </snapshots>
   </repository>
 </repositories>

naffan2014 · 2021-09-09T04:16:39Z

Add this in your pom.xml

<repositories>
   <repository>
     <id>oss.sonatype.org-snapshot</id>
     <url>https://oss.sonatype.org/content/repositories/snapshots</url>
     <releases>
       <enabled>false</enabled>
     </releases>
     <snapshots>
       <enabled>true</enabled>
     </snapshots>
   </repository>
 </repositories>

sorry

Add this in your pom.xml

<repositories>
   <repository>
     <id>oss.sonatype.org-snapshot</id>
     <url>https://oss.sonatype.org/content/repositories/snapshots</url>
     <releases>
       <enabled>false</enabled>
     </releases>
     <snapshots>
       <enabled>true</enabled>
     </snapshots>
   </repository>
 </repositories>

i use gradle ,not maven~ do u know how to add these in gradle ?

a-schild · 2021-09-09T06:07:33Z

Not sure, but probably something like this:

https://docs.gradle.org/current/userguide/declaring_repositories.html#sec:repository-content-filtering ?

naffan2014 · 2021-09-10T03:21:53Z

Not sure, but probably something like this:

https://docs.gradle.org/current/userguide/declaring_repositories.html#sec:repository-content-filtering ?

thank u . i can get 3.2.0-SNAPSHOT now

naffan2014 · 2021-09-10T05:29:27Z

when i use 3.2.0-snapshot . i can;t do the concat mp4 work again. issue is #166

a-schild · 2021-09-10T12:17:53Z

@naffan2014 Can we close this issue and continue on #166 ?

naffan2014 · 2021-09-10T14:57:09Z

@naffan2014 Can we close this issue and continue on #166 ?

yes。i close this issue at once.

naffan2014 closed this as completed Sep 10, 2021

a-schild mentioned this issue Feb 15, 2022

Cannot run program "/tmp/jave/ffmpeg-amd64-2.7.3": error=26, Text file busy #194

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fatal error when lots of ffmpeg process doing #163

fatal error when lots of ffmpeg process doing #163

naffan2014 commented Sep 3, 2021

a-schild commented Sep 3, 2021

naffan2014 commented Sep 3, 2021

a-schild commented Sep 3, 2021

naffan2014 commented Sep 3, 2021

naffan2014 commented Sep 3, 2021 •

edited

naffan2014 commented Sep 3, 2021 •

edited

a-schild commented Sep 3, 2021 •

edited

a-schild commented Sep 3, 2021

a-schild commented Sep 3, 2021

naffan2014 commented Sep 3, 2021

naffan2014 commented Sep 3, 2021

naffan2014 commented Sep 3, 2021

a-schild commented Sep 3, 2021 •

edited

naffan2014 commented Sep 9, 2021

a-schild commented Sep 9, 2021

naffan2014 commented Sep 10, 2021

naffan2014 commented Sep 10, 2021

a-schild commented Sep 10, 2021

naffan2014 commented Sep 10, 2021

fatal error when lots of ffmpeg process doing #163

fatal error when lots of ffmpeg process doing #163

Comments

naffan2014 commented Sep 3, 2021

a-schild commented Sep 3, 2021

naffan2014 commented Sep 3, 2021

a-schild commented Sep 3, 2021

naffan2014 commented Sep 3, 2021

naffan2014 commented Sep 3, 2021 • edited

naffan2014 commented Sep 3, 2021 • edited

a-schild commented Sep 3, 2021 • edited

a-schild commented Sep 3, 2021

a-schild commented Sep 3, 2021

naffan2014 commented Sep 3, 2021

naffan2014 commented Sep 3, 2021

naffan2014 commented Sep 3, 2021

a-schild commented Sep 3, 2021 • edited

naffan2014 commented Sep 9, 2021

a-schild commented Sep 9, 2021

naffan2014 commented Sep 10, 2021

naffan2014 commented Sep 10, 2021

a-schild commented Sep 10, 2021

naffan2014 commented Sep 10, 2021

naffan2014 commented Sep 3, 2021 •

edited

naffan2014 commented Sep 3, 2021 •

edited

a-schild commented Sep 3, 2021 •

edited

a-schild commented Sep 3, 2021 •

edited