[Bug?] Cannot deal with local files when run from command line via node

[binarykitchen]

Hello again

I'm not so sure if emcc generated javascript is able to act with local files.

For example I have this:

avconv> sudo node avconv.js -f image2 -r 15 -i ./tmp/images/%d.jpg -an -b:v 256k -loglevel info -c:v libvpx
avconv version v9-907-gf4b0d12, Copyright (c) 2000-2013 the Libav developers
  built on Apr  7 2013 22:20:42 with emcc (Emscripten GCC-like replacement) 1.3.0 (commit b22f6fbbbebb5df55ceb8fdc9f7c4d111c902c5e)
Trailing options were found on the commandline.
./tmp/images/%d.jpg: No such file or directory
...

you see it does not want to read files from the directory tmp/images within that folder. I opened the file avconv.js and tried to debug. But it's huge, 3.5 MB and full with heaps.

So I'm really unable to judge if it's a bug or not. Or maybe I have to change the location of the images or something? Any advice is very welcome.

Thanks,
Michael

[kripken]

Did you preload or embed the files you need?

https://github.com/kripken/emscripten/wiki/Filesystem-Guide

[binarykitchen]

@kripken interesting! i missed that out and have studied now. i will try to set up directories with 'preRun' ...

[binarykitchen]

@kripken okay here is my first problem: i cannot read the images. hope you can help?

i have added --pre-js avconv_pre.js and you can see its contents on that gist:
https://gist.github.com/binarykitchen/5329825

now when I run you see the error that it cannot read the images.

avconv> node avconv.jsavconv version v9-907-gf4b0d12, Copyright (c) 2000-2013 the Libav developers
  built on Apr  7 2013 22:20:42 with emcc (Emscripten GCC-like replacement) 1.3.0 (commit b22f6fbbbebb5df55ceb8fdc9f7c4d111c902c5e)
tmp/images/%d.jpg: No such file or directory

/home/michael.heuberger/projects/avconv/avconv.js:8832
      throw 'exit(' + status + ') called, at ' + new Error().stack;
                                               ^
exit(1) called, at Error
    at __exit (/home/michael.heuberger/projects/avconv/avconv.js:8832:50)
    at _exit (/home/michael.heuberger/projects/avconv/avconv.js:8834:7)
    at Array._open_input_file [as 556] (/home/michael.heuberger/projects/avconv/avconv.js:9750:187608)
    at _open_files (/home/michael.heuberger/projects/avconv/avconv.js:9750:179608)
    at _avconv_parse_options (/home/michael.heuberger/projects/avconv/avconv.js:9750:176712)
    at Object._main (/home/michael.heuberger/projects/avconv/avconv.js:9750:68040)
    at Object.callMain (/home/michael.heuberger/projects/avconv/avconv.js:11236:24)
    at doRun (/home/michael.heuberger/projects/avconv/avconv.js:11262:20)
    at run (/home/michael.heuberger/projects/avconv/avconv.js:11285:12)
    at Object.<anonymous> (/home/michael.heuberger/projects/avconv/avconv.js:11302:13)

the images are in a subfolder of that js file:
tmp/images/

what can i do to make these images readable? when i open line 9750 in vim to inspect Array._open_input_file i am struggling because code is huge, difficult to decipher. any hints?

[kripken]

You should be able to load the file normally from that path. Try to build a smaller case first. For example see test_files in tests/runner.py for an example. Also see other examples there with --preload and --embed.

[binarykitchen]

A smaller case? I cannot break down avconv in smaller pieces. And --preload / --embed is out of question for my case. I want to load files at runtime. So I'm using the FS object instead that loads all that at runtime.

1. Question

I have this in Module.preRun:

FS.createPath('/', 'tmp/images/', true, false);
FS.createPath('/', 'tmp/video/', false, true);

Tell me, how can I verify in a kind of debug mode if emcc really has set them up?

2. Question:

Does FS.createPath also load all the files within the given folder? According to the above example, does it also load tmp/images/1.jpg and tmp/images/2.jpg?

[kripken]

You can do FS.analyzePath('..') to check if a path is set up.

But createPath just creates the path. It doesn't know where to get contents for it. See emscripten_async_wget for a way to download files at runtime if preloading is not ok for you.

[binarykitchen]

Thanks for not giving up on me. I really want to solve this.

I tried FS.analyzePath('tmp/images/') and am getting this:

{ isRoot: false,
  exists: true,
  error: 0,
  name: 'images',
  path: '/tmp/images',
  object: 
   { read: true,
     write: false,
     timestamp: 1366089013023,
     inodeNumber: 10,
     isFolder: true,
     isDevice: false,
     contents: {} },
  parentExists: true,
  parentPath: '/tmp',
  parentObject: 
   { read: true,
     write: true,
     timestamp: 1366089013017,
     inodeNumber: 2,
     isFolder: true,
     isDevice: false,
     contents: { images: [Object], video: [Object] } } }

weird. contents are empty with {} but i have lots of jpg images in that folder. why aren't they listed here?

then i tried using emscripten_async_wget but get this error:
ReferenceError: emscripten_async_wget is not defined

how should i call this function properly?

then i tried something else with createPreloadedFile and found very interesting warnings. first, in preRun i added:

var source = 'tmp/images/';
var file = '1.jpg';
var url  = source + file;

var success = function() {
    console.log('success:');
    console.log(arguments);
};

var error = function() {
    console.log('error:');
    console.log(arguments);
};

FS.createPreloadedFile(
    source,
    file,
    url,
    true,
    false,
    success,
    error
);

then i see these warnings:

warning: no blob constructor, cannot create blobs with mimetypes
warning: no BlobBuilder
warning: cannot create object URLs

/home/michael.heuberger/projects/avconv/avconv.js:9564
        canvas.requestPointerLock = canvas['requestPointerLock'] ||
                                          ^
TypeError: Cannot read property 'requestPointerLock' of undefined
    at Object.Browser.init (/home/michael.heuberger/projects/avconv/avconv.js:9564:43)
    at Object.FS.createPreloadedFile (/home/michael.heuberger/projects/avconv/avconv.js:7944:17)
    at Array.Module.preRun [as 0] (/home/michael.heuberger/projects/avconv/avconv.js:20:4)
    at run (/home/michael.heuberger/projects/avconv/avconv.js:11422:15)
    at Object.<anonymous> (/home/michael.heuberger/projects/avconv/avconv.js:11474:3)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Function.Module.runMain (module.js:497:10)

what do these tell you?

[kripken]

emscripten_async_wget is a C function you call from the source C/C++. Otherwise, you should use Browser.asyncLoad.

[binarykitchen]

@kripken ok will check out Browser.asyncLoad this weekend. but what about my other questions? why were contents empty and what does no blob constructor, cannot create blobs with mimetypes mean in my case with createPreloadedFile()?

[kripken]

Contents should be empty until you place data in it with wget or another method.

No blob constructor means the browser lacks a crucial feature. Which browser and version are you on?

[binarykitchen]

forget the browser for now. i want to call it from command line.

i found a way to fill contents with readFileSync, see this new code:

var   fs = require('fs')
    , path = require('path');

var   source = 'tmp/images/'
    , destination = 'tmp/video/';

var Module = {
    'preRun': function() {
        var   sourcePath = FS.createPath('/', source, true, false)
            , destinationPath = FS.createPath('/', destination, false, true);

        var files = fs.readdirSync(source); 
        files.forEach(function(filename) {

            var file = path.join(source, filename);

            if (path.extname(filename) == '.jpg') { 

                var data = fs.readFileSync(file);
                FS.createDataFile(sourcePath, filename, data, true, false); 
            }
        });
    },

    'arguments': [
        '-f', 'image2',
        '-r', 15,
        '-i', source + '%d.jpg',
        '-an', // disable sound
        '-b:v', '512k',
        '-loglevel', 'info',
        '-c:v', 'libvpx',
        '-r', '25',
        '-y', destination + 'test.webm'
    ]
};

now FS.analyzePath(source) tells me that all images are loaded. great!
(do you think my above code is good or are there ideas for improvements?)

unfortunately, the big problem remains: avconv still says it cannot find any images in the folder:
avconv version v9-1055-gb71a050, Copyright (c) 2000-2013 the Libav developers built on Apr 20 2013 16:29:18 with emcc (Emscripten GCC-like replacement) 1.3.5 (commit 8408257cd66435af849f493c10c7f0e8d1d5fa3b) tmp/images/%d.jpg: No such file or directory

hmmm .... why?

[kripken]

Code looks ok. Make sure the program is looking at the exact same path you added the files in. Relative/absolute paths might be an issue.

[binarykitchen]

Absolute paths do not work either. When I prepend source and destination with /home/michael.heuberger/projects/avconv/ I get the very similar error message:
/home/michael.heuberger/projects/avconv/tmp/images/%d.jpg: No such file or directory

Do you want me to post the 3.3MB large avconv.js file for analysis? I have no idea what to do at this stage and need your help.

[kripken]

I suggest adding some debug printouts in the FS code - when it reaches analyzePath, print what it looks for and what does actually exist. It should call analyzePath from your C/C++ code that tries to read stuff.

[binarykitchen]

ok, i added a console.log(path); right after the function definition of analyzePath(...) and see that argument path is always / for every call.

not very helpful since i am expecting something with tmp/images in it.

i am preloading the images with createDataFile() but somehow the avconv code does not see it. i looked inside the js code and it looks very garbled, compact, difficult to deciper / debug that. all strings are memory references so really difficult to see what's wrong.

is there a way to compile the js file without having strings as memory references so that i can debug the code easier?

and do you know anyone else who could help me, really?

[kripken]

Compile with -g so it does not garble stuff and is easier to debug.

I recommend trying a smaller testcase. Try writing a standalone C file and load a file there, then read it from C.

[binarykitchen]

Code is less garbled with -g but still too difficult to debug. Lots of string variables still appear as numbers (memory references).

I do not know C well and can't do a smaller test case.

I really want to port ffpmeg/avconv to JavaScript very badly.I'm sure others want it too. Can you recommend anyone else who could support me?

[kripken]

I don't know, but you can ask on the mailing list or irc perhaps.

In general, ports like these do require a lot of C knowledge, enough to build and modify the original project. It's rare things work 100% out of the box.

[binarykitchen]

I see. Then I'm wiling to give an emscripten + C experienced person that job. @kripken Can you count me some names you know have been successful with emscripten and I'll ping them myself.

I already tried with the mailing list but noone replied. I really need help and want this to be solved. Thanks!

[kripken]

All I know are people that read the mailing list anyhow, so I doubt it would help (basically the same people you see on commits).

[binarykitchen]

Ok, will look at commits.

Tell me, did you develop emscripten alone?

[kripken]

I did most of the work originally, but it's gotten more balanced over time.

On Thu, May 2, 2013 at 6:39 PM, Michael Heuberger
notifications@github.comwrote:

Ok, will look at commits.

Tell me, did you develop emscripten alone?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1042#issuecomment-17374845
.

[binarykitchen]

Respect for what you did here!

Alright, I have asked emscripten-discuss@googlegroups.com for help. Again. Wait see if anybody responds. ...

[binarykitchen]

Still no response ... can anyone help me?

[stale]

This issue has been automatically marked as stale because there has been no activity in the past 2 years. It will be closed automatically if no further activity occurs in the next 7 days. Feel free to re-open at any time if this issue is still relevant.