Potential threading issue

[jamon]

If I try to clone or fetch multiple projects at the same it seems to fail silently sometimes, leaving the promise unresolved forever. I'm testing against a remote repo using the SSH protocol.

[johnhaley81]

@jamon can you run NodeGit.setThreadSafetyStatus(NodeGit.THREAD_SAFETY.ENABLED_FOR_ASYNC_ONLY); before you do the clone/fetches and see if it breaks?

[jamon]

Same issue. If I set it to clone 3 repositories, firing them all off at the same, at least one project will always clone successfully, but sometimes 1 or 2 of them will fail. I replicate it by removing the local copy and re-running the same script over and over, at least 1 of the 3 will fail about 50% of the time.

When it does fail, my credential and progress callbacks do not get called for the affected project, and the promise remains unresolved with no exceptions thrown.

[jamon]

Is there a connect timeout, clone timeout available? or perhaps the ability to cancel a connection cleanly so that one could implement their own timeout?

[jamon]

Sorry for the spam, one other useful piece of information is that if I kill the process after waiting a while, the resulting directory will have a valid .git folder and the remote will be added, but the fetch/clone against the origin never completed, so "git reflog" returns empty in the folder.

[johnhaley81]

/cc @srajko

[srajko]

@jamon thanks for the report. Which version of nodegit are you using?

[jamon]

Built from master, I can try a release build if you'd like

On Fri, Sep 2, 2016, 4:51 PM Stjepan Rajko notifications@github.com wrote:

@jamon https://github.com/jamon thanks for the report. Which version of
nodegit are you using?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1117 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAQqNRFhLF4gMBr5WvOEtFZ4HaTY9ugIks5qmJpEgaJpZM4Jzzo2
.

[srajko]

The number of threads that nodegit spins up is currently hard-coded to 10. You can try increasing that number (https://github.com/nodegit/nodegit/blob/master/generate/templates/templates/nodegit.cc#L122) to see if it makes a difference, but I think 10 threads should be enough for what you're doing.

If that doesn't help, could you please send a minimal script that reproduces the problem using public repos?

[jamon]

I did some more research and I'm unable to reproduce this against github, which leads me to believe that it is specific to something our internal repository is doing. My guess is that it's throttling requests or that we're hitting some other issue on that server establishing the ssh connection (I haven't tried non-ssh to see if it happens there).

Is there any way to cleanly set a timeout for fetches?--either as a timeout passed to the fetch, after which it would error, or by setting a timeout myself and cleanly cancelling the fetch request?

[jamon]

after more research, I've found that:

int _git_ssh_setup_conn in vendor/libgit2/src/transports/ssh.c never completes, I'm digging a bit deeper, but I'm not a C programmer, and I don't have a proper debug environment setup, so it's slow-going.

I'm currently adding printf's and re-running npm install on the whole nodegit module, if you have pointers on a better workflow for troubleshooting, I'm happy to give it a try.

[jamon]

More details....

in vendor/libgit2/src/transports/ssh.c, in the _git_ssh_setup_conn method:

    error = t->owner->certificate_check_cb((git_cert *) cert_ptr, 0, host, t->owner->message_cb_payload);

We get to this line, my callback code for the certificate check gets executed:

        certificateCheck: function() {
            console.log("node -- CERT CHECK");
            return 1;
        },

I see the console.log from it, but it libgit never gets past that line.

[maxkorp]

I'm currently adding printf's and re-running npm install on the whole nodegit module...

Welcome to my hell :/ lol.

FWIW, you can run node-gyp build if you have node-gyp globally installed, and that will cause the rebuild to be way faster than a full install.

[jamon]

Yeah, I did try doing that, but while it built, it didn't actually use my new code. At this point I'm throwing in the towel for now. I'm just going to wrap all of nodegit in my own wrapper, execute it via fork(), and send commands to/from my wrapper, with timeouts associated with them. Whenever nodegit/libgit/libssh hang forever like it's been doing, I'll just kill the process and recreate it.

It's a lot of work, but I can't risk nodegit causing my service to leak memory or calls to hang forever.

[DMXL]

It's a lot of work, but I can't risk nodegit causing my service to leak memory or calls to hang forever.

Unfortunately I suppose this issue still exists. After seeing this post I've decided to seek other solutions.

[implausible]

NodeGit should not be leaking memory anymore as of #1570. I'm not sure about hanging forever.