Error: EISDIR keeps on appearing when using downloadDir

[iMoses]

I'm using the downloadDir method and every once in a while, for no apparent reason but always on the same directories, I am getting this error: Error: EISDIR, open '/home/imoses/resources/index/FE filters/'.

I tried debugging it but I had no success.
Apparently graceful-fs tries to open a write stream on a directory path and I can't seem to understand why.

This is my code snippet.

    s3client.downloadDir({
        localDir: path.join(process.env.HOME, 'resources'),
        s3Params: {Bucket: '1k-resources', Prefix: 'db/'}
    }).on('end', cb)

Any help would be appreciated.

Running on:
NodeJS v0.10.25
Ubuntu 14.10

[iMoses]

Just in case anyone is interested in this, I ended up using getS3Params to check which path is a directory. In case of a file I execute the callback, otherwise I create the directory WITHOUT execute the callback to avoid this issue.

getS3Params: function(localFile, s3Object, callback) {
    if (path.extname(localFile) === '') { mkdirp.sync(localFile); }
    else { callback(null, {}); }
}

This check works for me since I have no files without an extension, but it can be easily modified to accept such cases as well.

[jujokini]

I had similar problem. For most of the buckets everything was working fine, but couple of them were failing with the EISDIR error code. I was able to fix it by setting "maxAsyncS3=1".

[bobzoller]

@iMoses solution did not work for me. I don't like the idea of never calling callbacks in general, so I also tried a version where I call the callback after mkdirp, but with null, null which AFAIK should skip the download. Still no luck.

@jujokini solution did work, but of course this means a huge reduction in speed.

[netzwerch]

@jujokini @bobzoller I can confirm that. It works, but the speed reduction is really a downside.

@iMoses I use "/" as delimiter for directories, and from the console output I saw that every directory had the "/" at the end. So I use the getS3Params to only skip the "download" of directories. That solved it for me.

getS3Params: function(localFile, s3Object, callback) {
    console.log("getS3params for: ", localFile);
    if (endsWith(localFile, "/")) {
        callback(null, null); // Skip folders
    } else {
        callback(null, {});
    }
}

function endsWith(str, suffix) {
    return str.indexOf(suffix, str.length - suffix.length) !== -1;
}

[faceleg]

@netzwerch do you think the code snippets you posted would go well in the new https://github.com/andrewrk/node-s3-client#examples section?

[netzwerch]

@faceleg Yes, it can definitely help people to work around this problem.
I assume that the error appears when creating a local writeStream on a "/some/path/" s3-key string. In the local filesystem (non-windows) this will be local folder name, so opening a stream on that causes the error. So maybe it could also be fixed inside of node-s3-client. But then again, why doesn't this happen with maxAsyncS3=1?

[bobzoller]

my assumption was it was a race condition -- that node-s3-client does the right thing with directory-looking-s3-keys like foo/bar/, but it also handles creating missing directories for file-looking-s3-keys like foo/bar/baz. if foo/bar/baz completes before foo/bar/, the latter blows up because the directory already exists.

[Piero87]

I get the same error, I have tried with this: maxAsyncS3=1, and resolve the problem, but the speed decrease. I have tried with @netzwerch solutions but do the same error. Any other help?

[Piero87]

I forgot the "/" at the end of my path, and with the @netzwerch solutions, it works. Thanks