Question: Error: EMFILE: too many open files #26

Newan · 2017-08-12T21:25:18Z

At first, thanks for this reader.

I want to read over 80k mp3 tags for my Software.
But after 7k the performance break down an finished with:

Error: EMFILE: too many open files....

Can someone help me to queue the async calls, with async or something like that:
https://caolan.github.io/async/docs.html#queue

Thanks

me code now is:

walker = walk.walk("/Volumes/..");
var mm = require('music-metadata');
const util = require('util');
walker.on("file", function (root, fileStats, next) {
 //todo, noch die anderen file extension freigeben
 if (path.extname(fileStats.name) === ".mp3") {
   mm.parseFile(path.join(root, fileStats.name)).then(function (metadata) {
     console.log(util.inspect(metadata, { showHidden: false, depth: null }));
   }).catch(function (err) {
     console.error(err.message);
   });
 }
 next();
});

The text was updated successfully, but these errors were encountered:

Borewit · 2017-08-13T08:13:09Z

next() should be called after metadata has been parsed.

walker = walk.walk("/Volumes/..");
var mm = require('music-metadata');
const util = require('util');
walker.on("file", function (root, fileStats, next) {
  //todo, noch die anderen file extension freigeben
  if (path.extname(fileStats.name) === ".mp3") {
    next(); // Queue (asynchronous call) parsing of metadata
    mm.parseFile(path.join(root, fileStats.name)).then(function (metadata) {
      console.log(util.inspect(metadata, { showHidden: false, depth: null }));
      next(); // asynchronous call completed, 'release' next 
    }).catch(function (err) {
      console.error(err.message);
    });
  } else { // this else is required
    next(); // Go to next file, no asynchronous call 'queued'
  }
});

So this is I guess how to fix your existing code.
I use a recursion pattern most of the time (not necessary better) to solve that problem:

var fileStats // Array of stats of directory listing. 

// Take MP3 files from file listing
var mp3Stats = fileStats.filter(function () {
  return path.extname(fileStats.name) === ".mp3";
});

function parseFiles (root, mp3Stats) {
  var mp3Stat = mp3Stats.shift();
  if (mp3Stat) {
    mm.parseFile(path.join(root, mp3Stat.name)).then(function (metadata) {
      console.log(util.inspect(metadata, { showHidden: false, depth: null }));
      parseFiles(root, mp3Stats); // recursion
    }).catch(function (err) {
      console.error(err.message);
    });
  }
}

Newan · 2017-08-13T09:25:06Z

Thanks for your feedback, great.

For your information:
I have Mp3-Files there are not finished in the function und not in the error case. This mp3 file return nothing. I testet is in a single file and no output was generated. tThis mp3 files are corrupted and cant play in any mediaplayer. Is there an option to skip this, like an open timeout or something?

My working try:

if (path.extname(fileStats.name) === ".mp3") {
  console.log("Mp3 found: " + path.join(root, fileStats.name));

  var timer;
  mm.parseFile(path.join(root, fileStats.name)).then(function (metadata) {
    //console.log(util.inspect(metadata, { showHidden: false, depth: null }));
    console.log("Metadaten found: next()");
    next();
    clearTimeout(timer);
  }).catch(function (err) {
    console.error(err);
    console.log("Error on reading metadata: next()");
    next();
    clearTimeout(timer);
  });

  timer = setTimeout(function () {
    console.log('Timeout on file: ' + path.join(root, fileStats.name));
    next();
    timer = null;
  }, 3000);
} else {
  console.log("No mp3() found");
  next();
}

if on Mp3 has an error it is always:

TypeError: Cannot read property 'trim' of undefined
    at Function.Common.parseGenre (/Volumes/CardReader/Projects/project-desktop/dist/node_modules/music-metadata/lib/common.js:115:28)
    at MusicMetadataParser.setCommonTags (/Volumes/CardReader/Projects/project-desktop/dist/node_modules/music-metadata/lib/index.js:210:46)
    at MusicMetadataParser.parseNativeTags (/Volumes/CardReader/Projects/project-desktop/dist/node_modules/music-metadata/lib/index.js:126:22)
    at /Volumes/CardReader/Projects/project-desktop/dist/node_modules/music-metadata/lib/index.js:89:26
    at <anonymous>

Edit:

After Error 7254 files the walker loop ends and the next file to open get: EMFILE: too many open files. But there are over 80k mp3 over. without reading metatags loop works.

Borewit · 2017-08-13T13:48:32Z

Regarding first problem: After Error 7254 files the walker loop ends:

I assume the too many open files are caused by massive parallel parsing of metadata. In principal the code above looks correct to me.

Maybe ensure sequence by:

console.log('Start parsing file: %s', fileStats.name);
mm.parseFile(path.join(root, fileStats.name)).then(function (metadata) {
      //console.log(util.inspect(metadata, { showHidden: false, depth: null }));
      console.log("Metadaten found: next()");
      console.log('Finish parsing file: %s', fileStats.name);
      next();
      clearTimeout(timer);
      })

Ensure output is like:

Start parsing file: A
Finish parsing file: A
Start parsing file: B
Finish parsing file: B

Point is, that A should finish entirely, before B is started.

Regarding second problem: issue parsing for ever in corrupt MP3:

mm.parseFile returns a promise.

A promise should always resolve (callback in then) or reject (callback in catch).
Providing it with a corrupted MP3 file (it is amazing what a crappy files are going around), is not an excuse for being idle forever.

This might be a bug in the music-metadata module, if you attach this MP3 file (preferably as a new issue), I will have a look what is going on.

Putting timers around this is not the way to fix it.

Borewit · 2017-08-13T19:37:11Z

Following code parsed 48k files without any issue:

const walk = require('walk')

var mm = require('music-metadata');
const util = require('util');
const path = require('path');

const walker = walk.walk("M:\\");

var filesParsed = 0;

walker.on("file", function (root, fileStats, next) {
  switch(path.extname(fileStats.name)) {
    case ".mp3":
    case ".flac":
      // Queue (asynchronous call) parsing of metadata
      mm.parseFile(path.join(root, fileStats.name)).then(function (metadata) {
        // console.log(util.inspect(metadata, { showHidden: false, depth: null }));
        console.log('Parsed %s files, last file: %s',  ++filesParsed, fileStats.name);
        next(); // asynchronous call completed, 'release' next
      }).catch(function (err) {
        console.error(err.message);
        next();
      });
      break;
    
    default:
      next(); // Go to next file, no asynchronous call 'queued'
  }
});

Newan · 2017-08-14T18:34:55Z

thanks for your code. I test it on an Macbook pro 8 gb Ram. Tomorrow i will test it on Windows10

Parsed 8111 files, last file: 16-Mandela Day.mp3
Parsed 8112 files, last file: Cure - Pictures Of You.mp3

<--- Last few GCs --->

  113464 ms: Mark-sweep 1197.7 (1436.0) -> 1197.5 (1436.0) MB, 1687.6 / 0.0 ms [allocation failure] [GC in old space requested].
  115158 ms: Mark-sweep 1197.5 (1436.0) -> 1197.5 (1436.0) MB, 1694.1 / 0.0 ms [allocation failure] [GC in old space requested].
  117005 ms: Mark-sweep 1197.5 (1436.0) -> 1205.7 (1413.0) MB, 1846.0 / 0.0 ms [last resort gc].
  118843 ms: Mark-sweep 1205.7 (1413.0) -> 1213.8 (1413.0) MB, 1838.1 / 0.0 ms [last resort gc].


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x31828cbcfb51 <JS Object>
    2: NewPromiseCapability(aka NewPromiseCapability) [native promise.js:175] [pc=0x1d7459bbc5c3] (this=0x31828cb04381 <undefined>,O=0x31828cbc3059 <JS Function Promise (SharedFunctionInfo 0x31828cb74f51)>)
    3: then [native promise.js:~215] [pc=0x1d7459bcd434] (this=0x2b0bdec68bb9 <a Promise with map 0xd7dfc885551>,z=0x2b0bdec68db9 <JS Function (SharedFunctionInfo 0x3c340ca8f2a1)>,A=0x31828c...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 ...

can you post an email or something to send you an mp3 that failed to load

Borewit · 2017-08-14T18:58:19Z

You should have a message with my email address in your inbox.

Borewit · 2017-08-19T15:22:48Z

@Newan: Can you try v0.7.15?

Newan · 2017-08-21T14:02:05Z

Version 0.8.0 works even better. if the mp3 collection is free from scrap all works fine. If not, you have an problem not the lib...

Thanks for the great support!

… handling multiple files.

Borewit added the question Question, clarification, discussion label Aug 13, 2017

Borewit self-assigned this Aug 13, 2017

Borewit mentioned this issue Aug 19, 2017

Empty / undefined tags in ID3v1 header are causing errors #30

Closed

Newan mentioned this issue Aug 19, 2017

Illegal ID3v2-tag-length throws 'Index out of range" exception #31

Closed

Newan closed this as completed Aug 21, 2017

Borewit mentioned this issue Aug 27, 2017

Parsing "MPEG-2 Audio Layer II" not working #36

Closed

Borewit mentioned this issue Mar 1, 2018

Slow duration parsing in some cases #58

Closed

Borewit mentioned this issue Aug 22, 2018

Synchronous metadata return #134

Closed

Borewit added a commit that referenced this issue Feb 2, 2019

#189, #134, #109, #26: Add Frequently Asked Question (FAQ) to explain…

c07aa87

… handling multiple files.

Borewit added a commit that referenced this issue Feb 2, 2019

#189, #134, #109, #26: Add Frequently Asked Question (FAQ) to explain…

14e3194

… handling multiple files.

Borewit mentioned this issue Feb 2, 2019

run function after processing multiple tracks #189

Closed

Borewit added a commit that referenced this issue Feb 2, 2019

#189, #134, #109, #26: Add Frequently Asked Question (FAQ) to explain…

4728e6b

… handling multiple files.

Borewit mentioned this issue Oct 16, 2019

Calling ignore with more then 8 MB will cause the application to exit Borewit/strtok3#55

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Question: Error: EMFILE: too many open files #26

Question: Error: EMFILE: too many open files #26

Newan commented Aug 12, 2017 •

edited by Borewit

Loading

Borewit commented Aug 13, 2017 •

edited

Loading

Newan commented Aug 13, 2017 •

edited by Borewit

Loading

Borewit commented Aug 13, 2017 •

edited

Loading

Borewit commented Aug 13, 2017 •

edited

Loading

Newan commented Aug 14, 2017 •

edited

Loading

Borewit commented Aug 14, 2017 •

edited

Loading

Borewit commented Aug 19, 2017

Newan commented Aug 21, 2017 •

edited

Loading

Question: Error: EMFILE: too many open files #26

Question: Error: EMFILE: too many open files #26

Comments

Newan commented Aug 12, 2017 • edited by Borewit Loading

Borewit commented Aug 13, 2017 • edited Loading

Newan commented Aug 13, 2017 • edited by Borewit Loading

Borewit commented Aug 13, 2017 • edited Loading

Borewit commented Aug 13, 2017 • edited Loading

Newan commented Aug 14, 2017 • edited Loading

Borewit commented Aug 14, 2017 • edited Loading

Borewit commented Aug 19, 2017

Newan commented Aug 21, 2017 • edited Loading

Newan commented Aug 12, 2017 •

edited by Borewit

Loading

Borewit commented Aug 13, 2017 •

edited

Loading

Newan commented Aug 13, 2017 •

edited by Borewit

Loading

Borewit commented Aug 13, 2017 •

edited

Loading

Borewit commented Aug 13, 2017 •

edited

Loading

Newan commented Aug 14, 2017 •

edited

Loading

Borewit commented Aug 14, 2017 •

edited

Loading

Newan commented Aug 21, 2017 •

edited

Loading