New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WebDAV sync broken in new versions (Android and MacOS) #624

Closed
kuyper opened this Issue Jun 15, 2018 · 36 comments

Comments

Projects
None yet
6 participants
@kuyper

kuyper commented Jun 15, 2018

Operating system

  • macOS
  • Android

Application

  • Desktop
  • Mobile

New version of Joplin on Mac (1.0.100) and on Android (1.0.127) do not sync over WebDAV anymore. It still works with the iOS version (10.0.24).

It seems that Joplin gets a 404 from my WebDAV server, but I could not find out what URL was being used in the debug mode or the logs.

I also used the function for testing the connection settings - also no luck.

The WebDAV server is working and I can access the correct URL with my webbrowser and log in, so the problem is very probably on the client side. And as mentioned above, the iOS version still works perfectly.

@kuyper

This comment has been minimized.

kuyper commented Jun 15, 2018

Just had a look into the server logs: it seems that Joplin does not even try to contact my webdav server. I see no access to the URL I had configured.

@kuyper

This comment has been minimized.

kuyper commented Jun 15, 2018

When I "check synchronisation settings" I get the following message:

Error. Please check that URL, username, password, etc. are correct and that the sync target is accessible. The reported error was:
WebDAV directory not found: () => { return typeof o[n] === 'function' ? on : o[n]; }

@kuyper

This comment has been minimized.

kuyper commented Jun 15, 2018

Just reverted to 1.0.99 on Mac and synchronisation works again.

@kuyper

This comment has been minimized.

kuyper commented Jun 15, 2018

Same is true for version 1.0.125 on Android

@laurent22

This comment has been minimized.

Owner

laurent22 commented Jun 15, 2018

Could you provide the log as described here? https://joplin.cozic.net/debugging/

@laurent22

This comment has been minimized.

Owner

laurent22 commented Jun 15, 2018

The only WebDAV change in this release was this one which seems somewhat related to what you're describing, but not sure what the issue is exactly as cannot replicate it. Do you happen to have a user I could test with on your server?

@laurent22

This comment has been minimized.

Owner

laurent22 commented Jun 21, 2018

Closing for now, but feel free to comment here if it's still happening.

@laurent22 laurent22 closed this Jun 21, 2018

@kuyper

This comment has been minimized.

kuyper commented Jun 21, 2018

I just tested with 1.101 on macos. Still doesn't work.

bildschirmfoto 2018-06-21 um 19 52 16

I did what is written in the debugging description but there is absolutely nothing in the console.

In log.txt I found this:

2018-06-21 19:51:43: "Starting scheduled sync"
2018-06-21 19:51:43: "Reducer action", "SYNC_STARTED"
2018-06-21 19:51:43: "Sync: starting: Starting synchronisation to target 6... [1529603503631]"
2018-06-21 19:51:43: "Reducer action", "SYNC_REPORT_UPDATE"
2018-06-21 19:51:43: "mkdir .sync"
2018-06-21 19:51:43: "mkdir .resource"
2018-06-21 19:51:44: "delta "
2018-06-21 19:52:03: "ResourceService::indexNoteResources: Start"
2018-06-21 19:52:03: "ResourceService::indexNoteResources: Completed"
2018-06-21 19:52:03: "ResourceService::deleteOrphanResources:", "[]"
2018-06-21 19:52:06: "Error: [object Object]
Code: 404
Error: [object Object]
at FileApiDriverWebDav.statFromResource_ (/Applications/Joplin.app/Contents/Resources/app/lib/file-api-driver-webdav.js:47:90)
at FileApiDriverWebDav.statsFromResources_ (/Applications/Joplin.app/Contents/Resources/app/lib/file-api-driver-webdav.js:116:22)
at FileApiDriverWebDav.list (/Applications/Joplin.app/Contents/Resources/app/lib/file-api-driver-webdav.js:277:22)
at
at process.tickCallback (internal/process/next_tick.js:188:7)"
2018-06-21 19:52:06: "Sync: finished: Synchronisation finished [1529603503631]"
2018-06-21 19:52:06: "Reducer action", "SYNC_REPORT_UPDATE"
2018-06-21 19:52:06: "Operations completed: "
2018-06-21 19:52:06: "Total folders: 7"
2018-06-21 19:52:06: "Total notes: 244"
2018-06-21 19:52:06: "Total resources: 31"
2018-06-21 19:52:06: "There was some errors:"
2018-06-21 19:52:06: "Error: [object Object]
Code: 404
Error: [object Object]
at FileApiDriverWebDav.statFromResource
(/Applications/Joplin.app/Contents/Resources/app/lib/file-api-driver-webdav.js:47:90)
at FileApiDriverWebDav.statsFromResources_ (/Applications/Joplin.app/Contents/Resources/app/lib/file-api-driver-webdav.js:116:22)
at FileApiDriverWebDav.list (/Applications/Joplin.app/Contents/Resources/app/lib/file-api-driver-webdav.js:277:22)
at
at process._tickCallback (internal/process/next_tick.js:188:7)"
2018-06-21 19:52:06: "Reducer action", "SYNC_COMPLETED"
2018-06-21 19:52:06: "Setting up recurrent sync with interval 300"
2018-06-21 19:52:06: "Updating all notifications..."
2018-06-21 19:52:06: "Garbage collecting alarms..."

@kuyper

This comment has been minimized.

kuyper commented Jun 21, 2018

Regarding the webdav server: it's a seafile seafdav behind an nginx reverse proxy. It worked with the older versions perfectly well, so I guess it's not a problem of that particular setup.

@kuyper

This comment has been minimized.

kuyper commented Jun 21, 2018

I'll be downgrding now because Joplin is beginning to become a really important part of my workflows... 😄

Thanks for that great piece of software!

@laurent22

This comment has been minimized.

Owner

laurent22 commented Jun 21, 2018

Ok I don't understand this error but it looks it might be due to the recent change for nginx (404 error handling). I'm going to disable this change for now and let's see in the next release if it works. CC @bradmcl

laurent22 added a commit that referenced this issue Jun 21, 2018

@kuyper

This comment has been minimized.

kuyper commented Jun 21, 2018

1.0.103 on macos works. Thanks!

@laurent22

This comment has been minimized.

Owner

laurent22 commented Jun 21, 2018

Nice, good to know SeaFile works again. @bradmcl, but it means Nginx is now broken again. Not sure how to fix it as I'm not able to test and don't know enough about it. However if you have a safe fix for Nginx feel free to let me know.

@bradmcl

This comment has been minimized.

Contributor

bradmcl commented Jun 21, 2018

Any chance we can get a peek at what that dav response from SeaFile looks like? Presumably there's a status line in there that contains a 404 - the context of what that is and why would be helpful.

@kuyper

This comment has been minimized.

kuyper commented Jun 21, 2018

What do you need and how can I obtain that info? Please understand that I am a bit reluctant to give out accounts on my server.

@bradmcl

This comment has been minimized.

Contributor

bradmcl commented Jun 21, 2018

Completely understand that! I feel the same way. What I'd like to see is the response of a PROPFIND request to your SeaFile server. One way to get that if you are comfortable at the command line is: curl -i --basic --user '<yourusername>' -X PROPFIND https://<yourservername>/<yourjoplinpath>/.sync --upload-file - -H "Depth: 1" <<end followed by the line end and your password at the prompt. Capture the output. Feel free not to share your username servername or joplinpath with us.

@Jestre

This comment has been minimized.

Jestre commented Jun 21, 2018

Here's a response from mine, if that helps:

`HTTP/2 207
server: nginx/1.13.12
content-type: application/xml
content-length: 714
date: Thu, 21 Jun 2018 22:11:52 GMT

<ns0:multistatus xmlns:ns0="DAV:">ns0:responsens0:href/seafdav/Joplin/.sync/</ns0:href>ns0:propstatns0:propns0:resourcetype<ns0:collection /></ns0:resourcetype>ns0:displayname.sync</ns0:displayname>ns0:getetag0000000000000000000000000000000000000000</ns0:getetag><ns0:lockdiscovery />ns0:supportedlockns0:lockentryns0:lockscope<ns0:exclusive /></ns0:lockscope>ns0:locktype<ns0:write /></ns0:locktype></ns0:lockentry>ns0:lockentryns0:lockscope<ns0:shared /></ns0:lockscope>ns0:locktype<ns0:write /></ns0:locktype></ns0:lockentry></ns0:supportedlock></ns0:prop>ns0:statusHTTP/1.1 200 OK</ns0:status></ns0:propstat></ns0:response></ns0:multistatus>%`

@bradmcl

This comment has been minimized.

Contributor

bradmcl commented Jun 21, 2018

Thanks! I think we're looking for one with a 404 hidden in it, but I'll add that one to a set of test cases for sure.

@Jestre

This comment has been minimized.

Jestre commented Jun 21, 2018

Is there a way to replicate the 404, e.g. ask for a file that doesn't exist? I am experiencing the error as well (haven't yet moved to 103).

@bradmcl

This comment has been minimized.

Contributor

bradmcl commented Jun 21, 2018

Sure, give that a try, just change .sync to doesnotexist or the like.

@laurent22

This comment has been minimized.

Owner

laurent22 commented Jun 21, 2018

I think the issue might be that the Nginx 404 detection code is in statFromResource_, which is used (among others) to list the WebDAV resources via PROPFIND /. Previously, if there was a 404 error in there, it would just ignore it, but now it's throwing an exception.

So I think to fix it we might need to either know why there's a 404 error code in the SeaFile response or, maybe better, to implement a fix for Nginx while still ignoring the 404 errors that can be ignored.

@laurent22

This comment has been minimized.

Owner

laurent22 commented Jun 21, 2018

Maybe try the above curl command but with the root "/" instead of a specific dir?

curl -i --basic --user '<yourusername>' -X PROPFIND https://<yourservername>/<yourjoplinpath>/ --upload-file - -H "Depth: 1" <<end

And then see if there's a 404 error somewhere in the result.

@Jestre

This comment has been minimized.

Jestre commented Jun 21, 2018

A la

server: nginx/1.13.12
content-type: text/html
content-length: 420
date: Thu, 21 Jun 2018 22:26:10 GMT

<!DOCTYPE HTML PUBLIC '-//W3C//DTD HTML 4.01//EN' 'http://www.w3.org/TR/html4/strict.dtd'>
<html><head>
  <meta http-equiv='Content-Type' content='text/html; charset=UTF-8'>
  <title>404 Not Found</title>
</head><body>
  <h1>404 Not Found</h1>
  <p>404 Not Found: The specified resource was not found</p>
<hr/>
<a href='https://github.com/mar10/wsgidav/'>WsgiDAV/1.2.0.pre</a> - 2018-06-21 22:26:10.069906
</body></html>%```
@bradmcl

This comment has been minimized.

Contributor

bradmcl commented Jun 21, 2018

Agreed, laurent, I'm trying to understand the SeaFile response a little better to craft in that direction.

Hmm, Jestre. That's not the magic response either that's running into trouble, since that would not have triggered the trace shown above. More thinking needed here. Thanks!

@Jestre

This comment has been minimized.

Jestre commented Jun 21, 2018

@laurent22 Doing it on the root ('/') yields a much longer response, since it appears to provide info for each file. I grepped through that, and there were no 404's except in the etag for a few of the files, so unless that is useful I'll save the space here :)

@bradmcl

This comment has been minimized.

Contributor

bradmcl commented Jun 21, 2018

Can you pastebin it or something? I would like to have it to test the next version of this fix.

@Jestre

This comment has been minimized.

Jestre commented Jun 21, 2018

@kuyper

This comment has been minimized.

kuyper commented Jun 22, 2018

Hi, I did this "curl -i --basic --user '' -X PROPFIND https:////.sync --upload-file - -H "Depth: 1" <<end" thing: (please keep in mind that the seafile server is behind an nginx reverse proxy)

HTTP/1.1 100 Continue

HTTP/1.1 207 Multistatus
Server: nginx/1.13.1
Date: Fri, 22 Jun 2018 06:14:48 GMT
Content-Type: application/xml
Content-Length: 630
Connection: keep-alive

<D:multistatus xmlns:D="DAV:"><D:response><D:href>/seafdav/Joplin/.sync/</D:href><D:propstat><D:prop><D:resourcetype><D:collection/></D:resourcetype><D:displayname>.sync</D:displayname><D:getetag>0000000000000000000000000000000000000000</D:getetag><D:lockdiscovery/><D:supportedlock><D:lockentry><D:lockscope><D:exclusive/></D:lockscope><D:locktype><D:write/></D:locktype></D:lockentry><D:lockentry><D:lockscope><D:shared/></D:lockscope><D:locktype><D:write/></D:locktype></D:lockentry></D:supportedlock></D:prop><D:status>HTTP/1.1 200 OK</D:status></D:propstat></D:response></D:multistatus>

@frbaroni

This comment has been minimized.

frbaroni commented Jun 23, 2018

Hello,

I'm having some sync issue with a flesh server/Nginx installation.
As it's a flesh installation, if needed, I can share the credentials. Simply email me.

I'm trying to sync with a Nginx webdav installation, it creates the .sync and .resource folders correctly, but it fails to create new files because now it fails to detect the file didn't exist (the 404 status issue).

I'm tracing the exception using the DevTools
I have this resource in file-api-driver-webdav.js:27

const result = await this.api().execPropFind(path, 0, [
	'd:getlastmodified',
	'd:resourcetype',
]);

Resource:

{
"$": {},
"d:response": [
  {
    "d:href": [
      "/notes/a5d68a18c39e4388baccc3df9c123379.md"
    ],
    "d:propstat": [
      {
        "d:prop": [
          "\n"
        ],
        "d:status": [
          "HTTP/1.1 404 Not Found"
        ]
      }
    ]
  }
]

}

Having the resource above, causes a crash after hitting the line

const lastModifiedString = this.api().resourcePropByName(resource, 'string', 'd:getlastmodified');

As d:getlastmodified returns null, WebDavAPI.js:166 crashes because it think the lastmodified is a non-null string tries to access it's index 0:

output = output[0];

What if we used the empty lastModified String to test if a file is a 404? :)

@bradmcl

This comment has been minimized.

Contributor

bradmcl commented Jun 24, 2018

So, when I go back and look at this again, I return to the original PR #541 that was rejected. It does do the narrowly defined fix for the call that NGINX makes.

In the light of what we now know, should that be the type of approach here? @laurent22

@laurent22

This comment has been minimized.

Owner

laurent22 commented Jun 25, 2018

@bradmcl, I don't know enough about the Nginx WebDAV implementation, but I get the feeling statFromResource_() is not the right place because this function is only to convert a WebDAV resource to a "stat" Joplin object, however with this change it's suddenly handling HTTP error codes, so that doesn't seem right.

Maybe post the XML that's causing an issue and we can try to figure out where to put the 404 detection code? Probably it should simply be in WebDavApi::exec() so that it can be detected early and passed properly to the error handling code.

@frbaroni

This comment has been minimized.

frbaroni commented Jun 25, 2018

Failing Nginx 404 response:

Request:

$ curl -i --basic --user 'user:pw' -X PROPFIND https://user@address.com/notes/ecd4027a5271483984b00317433e2c66.md --upload-file - -H "Depth: 1" <<end
<?xml version="1.0" encoding="UTF-8"?>
<d:propfind xmlns:d="DAV:">
	<d:prop xmlns:oc="http://owncloud.org/ns">
	        <d:getlastmodified />
	        <d:resourcetype />
	</d:prop>
</d:propfind>
end

Response

HTTP/2 207 
date: Mon, 25 Jun 2018 23:40:54 GMT
set-cookie: __cfduid=XXXXXX; expires=Tue, 25-Jun-19 23:40:54 GMT; path=/; domain=.address.com; HttpOnly
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
server: cloudflare
cf-ray: XXXXXX-GRU

<?xml version="1.0" encoding="utf-8" ?>
<D:multistatus xmlns:D="DAV:">
<D:response>
<D:href>/notes/ecd4027a5271483984b00317433e2c66.md</D:href>
<D:propstat>
<D:prop>
</D:prop>
<D:status>HTTP/1.1 404 Not Found</D:status>
</D:propstat>
</D:response>
</D:multistatus>

Response Result Object

{
  "d:multistatus": {
    "$": {},
    "d:response": [
      {
	"d:href": [
	  "/notes/ecd4027a5f78483984b00317433e2c66.md"
	],
	"d:propstat": [
	  {
	    "d:prop": [
	      "\n"
	    ],
	    "d:status": [
	      "HTTP/1.1 404 Not Found"
	    ]
	  }
	]
      }
    ]
  }
}
@laurent22

This comment has been minimized.

Owner

laurent22 commented Jun 28, 2018

Based on the XML above, I think the solution is to check if the multistatus contains only one response and, if it does, and that response contains a 404 status, throw an error.

It's a bit of a hack but if only Nginx does this, and only that response needs to be handled that would be fine for now. To make sure the error is caught early and handled properly, it should be in WebDavApi::exec().

I cannot test as I don't have an Nginx server running, but adding that code, below this line should work:

const responseArray = this.arrayFromJson(output, ['d:multistatus', 'd:response']);
if (responseArray && responseArray.length === 1) {
	const status = this.stringFromJson(output, ['d:multistatus', 'd:response', 0, 'd:propstat', 0, 'd:status', 0]);
	if (status && status.indexOf('404') >= 0) throw newError('Not found', 404);
}

If someone could test and confirm that would be great.

@Jestre

This comment has been minimized.

Jestre commented Jun 28, 2018

@laurent22 Do you need an account to test with?

@frbaroni

This comment has been minimized.

frbaroni commented Jul 7, 2018

@laurent22 , sorry for the delay, I could build & run with the recommended code, now it is able to create files and sync now! :)

@Phlogi

This comment has been minimized.

Phlogi commented Oct 23, 2018

This fix never was committed it seems. I do experience the same sync errors with nginx WebDAV. What's the status on this one?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment