Command hangs: mutagen sync terminate xxx

[rualark]

Example:

C:\D\mutagenmon\mutagen>mutagen sync terminate mushare
Terminating session 26458c28-3a17-4ae2-9af5-1f9d14c586ec...

Sometimes there is even no output for 10-20 seconds and only then "Terminating session" line appears

[xenoscopic]

I believe this is the same issue as #71. The issue is that the filesystem scan operations can block. I'm hoping to address this in v0.11.x. If we get accelerated scanning in place for Linux (perhaps via fanotify), that should also help prevent this in the majority of cases, but making scans preemptable is the correct fix.

[rualark]

This happens on windows host, which is also Alpha, sometimes when there is bad connection to Beta (which is Ubuntu).

For example, currently it is hanging and I have no way to end session. I tried to run mutagen sync terminate many times.

Is there other way to end session?

P.S. I was able to end session by killing mutagen process on Beta (Ubuntu), but only one session. I cannot end the second session.

[rualark]

State of session is Connected, Watching for changes.
Do you think that it has something to do with scan operation? In this case, why status is not Scanning?
Is there other way to end session? I rebooted to end session previously, but now I cannot end it again.

I have two different sessions and I cannot end any of them without reboot. I could end third session though.

[rualark]

I tried:
mutagen daemon stop
Then I could not run mutagen sync list with error Attempting to start Mutagen daemon... Error: unable to connect to daemon: connection timed out (is the daemon running?)

Then I tried mutagen daemon start
But yet I could not start mutagen sync list with the same error.

[rualark]

Same problem today: no way to terminate session. command hangs:

mutagen sync terminate 123
Started Mutagen daemon in background (terminate with "mutagen daemon stop")
Terminating session feca40ce-710b-4ddc-89c4-161b67962c67...

mutagen daemon stop and mutagen daemon start does not help, outputs nothing

[rualark]

Do you think the problem is with scan? I get this issue everyday now. And sessions have status 'Waiting for changes'.

[xenoscopic]

Hey @rualark, I may have closed this prematurely. Can you try v0.11.2 and see if you’re still seeing these hangs? I’ve turned on ServerAliveInterval and ServerAliveCountMax by default for all SSH commands, which I think may fix this issue. There was a code path where a transport command (like ssh or docker) might not realize that its transport had been terminated, and if a synchronization session was polling (“Waiting for changes”) and then manually cancelled, it was possible for the session to become hung indefinitely.

[rualark]

Thanks, I'll try

[rualark]

I got this so far. Will try to restart:

[rualark]

Ok, I restarted with no success:

Will remove .mutagen folder and try to restart again

Is there an upgrade guide? Probably I have some conflicts with previous 10.0 version?

[rualark]

I removed folders from .mutagen profile folder except daemon/daemon.lock and restarted again. After restart I see that some folders were recreated in .mutagen profile folder. I cannot delete daemon/daemon.lock file, although mutagen.exe process is not listed in running processes or services. How can this be?

[rualark]

Now I disabled mutagen.exe in Task manager/Startup, restarted and removed everything in .mutagen profile folder. Seems to work now.

[rualark]

This command seems to help to kill mutagen.exe process, which is not visible in Task manager:
taskkill /f /im mutagen.exe

Why is mutagen.exe is not visible in Task manager?

[rualark]

I migrated from Windows 10 to Ubuntu 18.04 approximately at the same time as I migrated to mutagen 0.11.2
I did not have any of the described issues so far. Mutagen is significantly more stable and about 5-10 times faster on Ubuntu.

I also adapted MutagenMon to run on Ubuntu 18.04 and will push later.

[edgar]

I'm experiencing a similar behavior with Mutagen v0.11.7 and Docker both v2.3.0.5 (48029) and v2.4.0.0 (48506) in Mac.

mutagen project terminate hangs while mutagen sync list reports the following:

❯ mutagen sync list
--------------------------------------------------------------------------------
Name: se
Identifier: sync_ocbLpeHyBsuw2oZywzaNJNvMYjzonTdfozMcI5qNSRO
Labels:
	io.mutagen.project: proj_LDWYCoWXeEkFq8SsJWOm9ue5ydKJTCFXzm5usPcwK3l
Alpha:
	URL: /Users/edgarg/dev/se
	Connection state: Connected
Beta:
	URL: docker://mutagen/app
		DOCKER_HOST=
		DOCKER_TLS_VERIFY=
		DOCKER_CERT_PATH=
		DOCKER_API_VERSION=
	Connection state: Connected
Status: Watching for changes
--------------------------------------------------------------------------------

In wardenenv/warden#235 people reports that downgrading to Docker v2.3..0.4 fixes the issue.

Anyone else experiencing the same issue? (I haven't tried downgrading Docker to v2.3.0.4)

[xenoscopic]

@edgar This was probably the same underlying issue as in #223. I've just back-ported the fix to v0.11.8, so hopefully that fixes the problem, but please let me know if not.

[b-sharpe]

@havoc-io I can confirm 0.11.8 worked for me on docker 2.4.0.0 on mac 10.15.7.

I did first have to remove each hung sync session manually, now everything works as expected.

[edgar]

@havoc-io I can confirm too that Mutagen v0.11.8 is working well with docker v2.4.0.0, thanks!