Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ffmpeg.decode_audio cannot be run in parallel #5804

Closed
jheymann85 opened this issue Nov 23, 2016 · 6 comments
Closed

ffmpeg.decode_audio cannot be run in parallel #5804

jheymann85 opened this issue Nov 23, 2016 · 6 comments
Assignees

Comments

@jheymann85
Copy link

To decode the input, the function writes the content to a temporary file. Its name is generated by the function GetTempFilename found in tensorflow/contrib/ffmpeg/default/ffmpeg_lib.cc. The template for the filename is %tmp_dir/tmp_file_%PID.%EXT.

When using multiple decoders in parallel this causes an undetermined behaviour since all decoders want to write and afterwards delete the same file.

A possible solution would be to use the thread id instead of the process id. I.e.

#include <sys/syscall.h>
#define gettid() syscall(SYS_gettid)
...
return io::JoinPath(dir, StrCat("tmp_file_", gettid(), ".", extension));

The first two lines are necessary because glibc does not wrap the call.

This solution works for me (on Linux). I'm, however, not sure if it works on all supported platforms. If that's fine, I can make a pull request.

@prb12
Copy link
Member

prb12 commented Nov 23, 2016

@fredbertsch You are listed as owner for this contrib dir... could you please take a look?

@jonasrauber
Copy link
Contributor

I also have this problem. Will you fix this?

@gunan
Copy link
Contributor

gunan commented Jun 16, 2017

@fredbertsch Any updates here?
Is this still a problem?

@rryan
Copy link
Member

rryan commented Oct 23, 2017

Just keeping folks in the loop, we have a candidate fix internally that @fredbertsch authored. Hopefully it will show up in master over the next day or two!

@fredbertsch
Copy link
Contributor

A test and fix were added internally, and they should propagate here soon.

benoitsteiner pushed a commit to benoitsteiner/tensorflow that referenced this issue Oct 25, 2017
Bug is at:

tensorflow#5804

Fix is to add a unique identifier to each temp file name. The id is unique to
the process. Multiple processes could still have a conflict, though even there
the odds do go down somewhat with this fix.

PiperOrigin-RevId: 173261202
@tensorflowbutler
Copy link
Member

It has been 14 days with no activity and this issue has an assignee.Please update the label and/or status accordingly.

@gunan gunan closed this as completed Dec 22, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants