Login & Sync errors in iOS

[ludwa6]

Since you closed Issue 313 Jamie -rightly so i guess, since the specific problems i reported seem to have been fixed in subsequent release(s)- i'm creating a new issue for this slightly different sync error i've encountered with version 0.4.15 , which went like this:

1. Create a new Activity log in farmOS (web interface), resetting date to future timing, to serve as ToDo in Field Kit;
2. Sync FieldKit, and open that new Log entry; then
3. Click into this item, and append new text into Comment field; then
4. Back on the MyLogs page, try to sync...

...Whereupon app threw the error message:
Error while syncing "(name of Log entry)"... The website encountered an unexpected error. Please try again later.

Now it doesn't matter how many times i "try again later," the server apparently cannot digest this FK-edited Log entry (although i manually reverted status of comment field to its pre-edit state, as it remained on server), so i was left with no choice but to delete the Log entry in FK... And subsequent sync would not retrieve the original item from server.

So i opened that Log entry which remained on server, repeated the edit i had done to Comment field in FK and saved, went back to FK and synced... And it pulled the edited item back into MyLogs page, no problem (giving me what i wanted in the end, but still...)

If all records should be editable and sync-able on either side (web interface or FK), then this is clearly a bug in the sync logic... And if certain edits are not permitted in FieldKit, then it's a UI issue, i.e.: the app should not allow the user to make them in the first place, which would avoid the condition of having the same record in two different states on server and mobile app, and unable to sync.

[jgaehring]

I haven't been able to reproduce this strictly with the steps you provided. You're experiencing this in the Android version?

[yenzand]

@ludwa6 and @jgaehring I have experienced the similar issues -

1. Changed the date on an existing activity log in field kit and got the same error and error message as @ludwa6 - then deleted the Log entry in field kit any subsequent syncs would not retrieve the original item from server. Original log entry still listed in farmOS.
2. Changed the date on an existing activity log in farmOS after syncing quite a few times in field kit the date is not updated. Also tried changing the log name - this also does not sync through to field kit.

I have a farmier subscription and using the Android app.

UPDATE - I logged out of field kit, logged back in and re-synced and now logs have updated in field kit. I also forward dated another existing activity log in farmOS and the log updated in field kit. Looks like logging back in corrected the issue/s.

[ludwa6]

To your Q, @jgaehring : my platform is iPhone 7, running iOS v-13.3.1 .

Just to confirm: i have now repeated the process with a new entry, and replicated the error.

So: instead of just deleting the problematic entry as before, i tried @yenzand 's solution of logging out and logging in, pushing the sync button... And the record came back from server as it was, without the edited comment i had just done in FK.

NB: This was the ONLY log that came back into MyLogs list (which was cleared to empty state by the logout) -a problematic condition, because FK had contained not only a list of Logs in "Done" state, but also the "Not Done" item at issue in the beginning of this thread. All those items had been previously synced (so maybe that's why the algorithm "thought" it didn't need to resync?)... But this is clearly at-odds with the "FK as ToDo List Manager" UseCase.

[jgaehring]

UPDATE - I logged out of field kit, logged back in and re-synced and now logs have updated in field kit. I also forward dated another existing activity log in farmOS and the log updated in field kit. Looks like logging back in corrected the issue/s.

Hm, this makes me think this is related to the issue we faced where the server was sending numbers as strings. We recently changed how we dealt with that in b2592ba. My guess is this is a one-off problem that was caused by some sort of clash between old and new data being in different formats.

And the record came back from server as it was, without the edited comment i had just done in FK.

This is an unfortunate but expected result, if you logged out before a sync successfully completed.

NB: This was the ONLY log that came back into MyLogs list (which was cleared to empty state by the logout) -a problematic condition, because FK had contained not only a list of Logs in "Done" state, but also the "Not Done" item at issue in the beginning of this thread. All those items had been previously synced (so maybe that's why the algorithm "thought" it didn't need to resync?)... But this is clearly at-odds with the "FK as ToDo List Manager" UseCase.

Are you saying that logs that had been in a "Not Done" state were not synced to FK after logging back in?

[ludwa6]

Are you saying that logs that had been in a "Not Done" state were not synced to FK after logging back in?

Yes, @jgaehring , that's is what i'm saying: that's the problem i see with the logout/login workaround, to the more fundamental problem of sync process choking on these Log entries created on desktop and subsequently edited in FK.

[jgaehring]

@ludwa6, can you verify that those logs are still marked as "Not Done" on the server?

[ludwa6]

Yes, @jgaehring , confirmed... But now FK has gone totally inoperative on me. The one log entry i synced yesterday was gone from MyLogs screen, leaving it empty. Tried to sync, and nothing came thru. Tried to create a new Log entry, and got a blank screen -no change over many minutes. Logged out and back in; no change. Deleted the app, and tried to reinstall from TestFlight app/ invite link... And the install failed (first time this has ever happened). So: i dunno what to do next. ?

[jgaehring]

Sorry I'm just getting a chance to follow up with this now.

@ludwa6, you and I spoke in chat briefly to try to debug this, and I suspect that these might be particular to the native iOS build. You said you did not experience the same errors when using the PWA at https://farmos.app, correct?

[ludwa6]

Sorry @jgaehring , but no. To be clear, it's like this:

• web interface to my instance (vdl.farmos.net) works fine, on mobile as well as desktop, but
• PWA pulls a "unable to reach the server" error at same URL; and
• FieldKit app remains broken (even after reinstall) as reported above.

NB: My mobile platform is iPhone 7 running iOS 13.3.1, with FieldKit (version 0.4.15) installed via TestFlight.

[jgaehring]

I was able to test the PWA on desktop Safari today and on login I got the same error. It appears that the cookies needed for authentication are not being sent or received by Safari. This is one of the few threads I found that seems to address a similar issue: https://elixirforum.com/t/set-cookie-header-works-on-google-chrome-and-ff-but-no-safari/20867. If the thread is correct and applies to our situation, we may be out of luck, unless we were to bundle a version of Field Kit with every farmOS server instance so it could be served from the same domain.

The only other possible source of failure I can find would be in farmOS.js:

https://github.com/farmOS/farmOS.js/blob/fb4f727fe32d2728b87d2342d368073a7ff17b30/index.js#L30-L37

I've ruled this out because, from what I can tell, Safari is still performing the redirect from user/login to users/tester. Here's a complete dump of the request/response headers from both Ubuntu Chrome and MacOS Safari for comparison:

UBUNTU CHROME

General

Request URL: https://test.farmos.net/user/login
Request Method: POST
Status Code: 302 Found
Remote Address: 34.195.85.140:443
Referrer Policy: no-referrer-when-downgrade

Response Headers

Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: Content-Type,Authorization,X-CSRF-Token
Access-Control-Allow-Methods: GET,POST,PUT,DELETE,HEAD,OPTIONS
Access-Control-Allow-Origin: https://farmos.app
Cache-Control: no-cache, must-revalidate
Connection: keep-alive
Content-Length: 0
Content-Type: text/html; charset=UTF-8
Date: Sun, 22 Mar 2020 15:58:27 GMT
Expires: Sun, 19 Nov 1978 05:00:00 GMT
Location: https://test.farmos.net/users/tester
Server: nginx/1.17.8
Set-Cookie: SESS9646fa17283c8d6f6c92f39f1c0e12b4=nEqqc8_b65KPw4Ncff7HJkBMWRT5uSHaJmV2o6tuQf4; expires=Tue, 14-Apr-2020 19:31:47 GMT; Max-Age=2000000; path=/; domain=.test.farmos.net; HttpOnly
Strict-Transport-Security: max-age=31536000
X-Content-Type-Options: nosniff

Request Headers

Accept: json
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Connection: keep-alive
Content-Length: 44
Content-Type: application/x-www-form-urlencoded
Cookie: Drupal.tableDrag.showWeight=0; has_js=1
Host: test.farmos.net
Origin: https://farmos.app
Referer: https://farmos.app/
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: cross-site
User-Agent: Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Mobile Safari/537.36

General (Redirect)

Request URL: https://test.farmos.net/users/tester
Request Method: GET
Status Code: 200 OK
Remote Address: 34.195.85.140:443
Referrer Policy: no-referrer-when-downgrade

Response Headers (Redirect)

Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: Content-Type,Authorization,X-CSRF-Token
Access-Control-Allow-Methods: GET,POST,PUT,DELETE,HEAD,OPTIONS
Access-Control-Allow-Origin: https://farmos.app
Cache-Control: no-cache, must-revalidate
Connection: keep-alive
Content-Encoding: gzip
Content-Language: en
Content-Length: 3907
Content-Type: text/html; charset=utf-8
Date: Sun, 22 Mar 2020 15:58:27 GMT
Expires: Sun, 19 Nov 1978 05:00:00 GMT
Server: nginx/1.17.8
Strict-Transport-Security: max-age=31536000
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Generator: Drupal 7 (http://drupal.org)

Request Headers (Redirect)

Accept: json
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Connection: keep-alive
Content-Type: application/x-www-form-urlencoded
Cookie: Drupal.tableDrag.showWeight=0; has_js=1; SESS9646fa17283c8d6f6c92f39f1c0e12b4=nEqqc8_b65KPw4Ncff7HJkBMWRT5uSHaJmV2o6tuQf4
Host: test.farmos.net
Origin: https://farmos.app
Referer: https://farmos.app/
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: cross-site
User-Agent: Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Mobile Safari/537.36



MACOS SAFARI

Summary

URL: https://test.farmos.net/user/login
URL: https://test.farmos.net/users/tester
Status: 403 Forbidden
Source: Network
Address: 34.195.85.140:443
Initiator:
xhr.js:160

Request

GET /user/login
Accept: json
Content-Type: application/x-www-form-urlencoded
Origin: https://farmos.app
Referer: https://farmos.app/
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.5 Safari/605.1.15

Redirect Response

302 Found
Access-Control-Allow-Origin: https://farmos.app
Access-Control-Allow-Methods: GET,POST,PUT,DELETE,HEAD,OPTIONS
Expires: Sun, 19 Nov 1978 05:00:00 GMT
Access-Control-Allow-Headers: Content-Type,Authorization,X-CSRF-Token
Cache-Control: no-cache, must-revalidate
Date: Sun, 22 Mar 2020 15:34:21 GMT
Location: https://test.farmos.net/users/tester
Access-Control-Allow-Credentials: true

Request

POST /users/tester HTTP/1.1
Accept: json
Content-Type: application/x-www-form-urlencoded
Origin: https://farmos.app
Host: test.farmos.net
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.5 Safari/605.1.15
Referer: https://farmos.app/
Connection: keep-alive

Response

HTTP/1.1 403 Forbidden
Content-Type: text/html; charset=utf-8
Access-Control-Allow-Origin: https://farmos.app
Access-Control-Allow-Methods: GET,POST,PUT,DELETE,HEAD,OPTIONS
Expires: Sun, 19 Nov 1978 05:00:00 GMT
Transfer-Encoding: Identity
Access-Control-Allow-Headers: Content-Type,Authorization,X-CSRF-Token
Content-Language: en
Cache-Control: no-cache, must-revalidate
Date: Sun, 22 Mar 2020 15:34:21 GMT
Access-Control-Allow-Credentials: true
Connection: keep-alive
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
Server: nginx/1.17.8
X-Generator: Drupal 7 (http://drupal.org)

[jgaehring]

Ok, I think I've found a possible solution for the PWA login issue, although not ideal.

By default, Safari blocks cross-site cookies, but this setting can be disabled by going to Safari > Preferences > Privacy, then deselecting "Prevent cross-site tracking". More info on this setting can be found here: https://support.apple.com/guide/safari/prevent-cross-site-tracking-sfri40732/13.0/mac/10.15. When I deselected this option, the login process finally worked.

Regarding the blank screen when trying to create a new log, I'm now able to produce this error in Safari as well. Debugging now and will get back to you with an update.

[jgaehring]

Regarding the blank screen when trying to create a new log, I'm now able to produce this error in Safari as well. Debugging now and will get back to you with an update.

The source of the issue appears to be that IndexedDB is not able to open a cursor, which causes the generation of the log's localID to fail:

I did a little bit of searching but could only find a couple seemingly related bugs in other projects:

• Cannot open cursor on iOS with 'nextunique' direction jakearchibald/idb#71
• Unable to open cursor when submitting changes in Safari periodo/periodo-client#228

I have to put this down for now, not sure when I'll get a chance to debug on a Mac, but will keep y'all posted.

[jgaehring]

@ludwa6 I created a branch preview to test out a hunch:

https://deploy-preview-329--dreamy-tesla-8069e3.netlify.com/

If you have a chance, would you mind trying out the preview to see if it works in iOS and/or macOS?

I don't anticipate this will resolve the login issue (see #328), but try opening a new log and see if the blank screen still appears. It's a long shot, might not work, but it's a quick'n'easy fix that I can push out to the native iOS app if it works there.

[ludwa6]

@jgaehring : Tried on both iOS and Mac, with same result:

• CAN access the form for adding a new log, but
• can NOT save nor sync, because login falls with the "unable to reach server" message.

So, a bit of progress here... But until the server access issue is fixed, FieldKit remains inoperative for me, bottom-line.

[jgaehring]

Ok, that's all good, and as expected. As I mentioned in #328 (comment), the login problem should only effect the PWA in iOS, not the native iOS. So we only have to push these changes out to TestFlight and you should have a working Field Kit again. 🙂

Thanks for all your help in debugging this, I really appreciate it!

[jgaehring]

Just a heads-up, @ludwa6, I submitted v0.4.16 for review with the App Store just now, so you should see an update sometime tomorrow or Friday at the latest. Hopefully this will resolve the issue with the + icon. If you're still getting issues with syncing, we can reopen this issue.

Also, the whole reason this release was held up was because I had to completely overhaul the app building process (#323) due to a change in Apple's guidelines. It's the kind of overhaul I'd only make if my hand was forced (which it was), so if you get a chance to try it out, and have a minute to report back that it's not crashing horribly, I'd really appreciate it. 😉

[ludwa6]

Received and installed the update this morning, @jgaehring ; it seems a partial success in that clicking the '+' button does take me to the 'Edit Log' form (so the "white screen" problem is solved), but the overall problem here remains. Sync still doesn't work -and now the "Add my GPS Location to the Log" button doesn't work[1]- so the app is effectively inoperative for me.

[1] I did check the "Share My Location" switch on startup, in case you were wondering, and it is enabled.

[jgaehring]

When you say sync doesn't work, I assume that it's still displaying the same "unable to reach server" error when you try to login? If that's the case, then that, combined with the fact that GPS isn't working either, makes me think it's something to do with network connectivity. It might be that, even if other apps on your phone are able to connect to the network, something is broken (or getting blocked) in the way that Field Kit specifically connects to the network, perhaps related to #320.

I think it would be helpful at this point to see if other iOS users are experiencing the same problem, because I am unable to reproduce this error on any devices available to me, either in the Android app or the browser. I'll try to ask around to folks I know and on Riot, and if there's anyone else on your farm with an iPhone, it wouldn't hurt to see if they get the same problem logging in.

[jgaehring]

To any iPhone or iPad users directed here for testing, please try the following steps* to reproduce this error:

• Download or update the farmOS Field Kit app in Test Flight (more instructions here).
• Confirm that you have the latest version, 0.4.16, by opening the left-hand menu; it should appear at the bottom.
• In the same menu, go to "Login"
• Enter your farmOS url (eg, test.farmos.net), your email or username, and your password, then click "Submit credentials".
• If the login succeeds, you should be redirected to the "My Logs" screen.
• If the login fails, an error message should be displayed at the top of the Login screen.
• Let us know below whether the login succeeded or failed, and if it failed, what the error message says. If you can, please also include which version of iOS you currently have installed on your device.

If login does fail, there's one other crucial test you could try for us:

• Open Test Flight again and go to farmOS Field Kit's app page.
• Tap on "Previous Builds".
• Tap and install version 0.4.15; this will rollback changes to the previous version.
• Go back to Field Kit, and verify in the main menu that it's now at 0.4.15
• Tap "Login" again, and try logging in.
• Let us know below if login succeeds. (NB: other issues may still persist, such as logs' quantities in app field not registered #330, and/or the white screen mentioned in comments above; those are to be expected and can be ignored.)

Thanks!

* @ludwa6, please correct me if I got anything wrong in these steps to reproduce.

[jgaehring]

After some discussion with @mstenta, I think we've identified 3 possibilities for what could be causing the login to fail:

1. Apple has deprecated UIWebView sooner than they announced (see iOS: Migrate from UIWebView to WKWebView #320), and this has broken essential browser API's such as XMLHttpRequest and Geolocation.
2. Contrary to what I thought, commit c549d7b did succeed in activating the WKWebView engine, (even though it still triggered Apple's deprecation warning; see Still getting "UIWebView" warning when compiling to iOS through Phonegap Builder apache/cordova-plugin-wkwebview-engine#125 (comment)); also, WKWebView is treating XHR requests from the file:// protocol as CORS requests, as per this notice; therefore, Apple's prohibition against cross-site cookies is breaking our authentication procedure the same way it is for the PWA (see Login fails in Safari on iOS/macOS #328).
3. Somehow Apple's cross-site cookie policy is breaking our auth procedure even under UIWebView.

No.1 should be easy to test with updated procedure I posted in #324 (comment) for rolling back to 0.4.15. If login still fails in 0.4.15, then No.1 seems like our most likely culprit, and we need to push out a release that successfully migrates us to WKWebView to see if that alone fixes it.

If login succeeds in 0.4.15, then we can rule out No.1 and assume No.2 to be the most likely culprit. In that case, I propose we take the following steps:

• Prepare an 0.4.17 release, which just reverts the WKWebView plugin, but keeps the fixes for quantities and the white screen of death, and get that pushed out before April 1. Hopefully that will buy us time for the next steps, since Apple is only disallowing new submissions starting in April, and hopefully won't deprecate UIWebView in older submissions.
• Keep a branch that retains the WKWebView commit, and add the necessary configuration to make sure it no longer triggers the deprecation warning from Apple (see iOS: Migrate from UIWebView to WKWebView #320 (comment)).
• Start implementing OAuth (Authenticate via OAuth2 #311) ASAP, so once we are forced to release with WKWebView, we have an authentication procedure in place that will work with it.

No.3 seems least likely of all, and should be a moot point once we've implemented the solution for No.2.

I'm going to wait to hear back from some users about No1, but in the meantime will prepare 2 separate branches, which we can release ASAP once we hear if login works in 0.4.15: one branch that reverts commit c549d7b, which added the WKWebView plugin; and one branch which keeps that commit and makes sure the WKWebView is fully activated and that UIWebView is no longer referenced.

[ludwa6]

Am not sure what more i can do, @jgaehring -i have no other iOS users to engage for testing here, they all use Android- but i can confirm that it's the same "unable to reach server" error msg that i get, when i try to login w/ version 0.4.16. Beyond that i guess i can only wait for a new version.

[jgaehring]

Hey @ludwa6! Did you try the rollback instructions to test whether login work if you you go back to 0.4.15? This instructions are at #324 (comment)

I just need to confirm for sure whether or not authentication works, even if syncing doesn't, before I move forward with next steps.

[jgaehring]

I just realized, in reference to #324 (comment) above, there is also the possibility that both numbers 1 and 2 are true; that UIWebView got deprecated at some point, and so is breaking the XHR API in releases <= 0.4.15, while at the same time, 0.4.16 activated WKWebView and that it is broken because of CORS. So if we merge in #335, and it's still not working, our next goal should be to get OAuth implemented as quickly as possible.

[jgaehring]

Another thing we can do is release a debugging build but only distribute it to a select few people to test it and ask them to take screenshots of the debugging info. Some examples of info we (might) want to collect:

• navigator.connection (via cordova-plugin-network-information*)
• navigator.onLine
• navigator.userAgent
• navigator.platform
• navigator.cookieEnabled
• "XMLHttpRequest" in window
• "Geolocation" in window
• "WKWebView" in window

* the Network Information API is unsupported in Safari 😠

[jgaehring]

Version 0.4.17 is now released to the App Store. It hopefully reverts the changes that broke login/sync, but retains the changes that fixed the white screen of death. If you have a chance to test, please let us know your experience!

[ludwa6]

Sorry to say, @jgaehring : tho it does retain the fix to whitescreen problem, i still can't log in with new version 0.4.17. Have tried reboot, reinstall, etc... No way around this that i can find on my end.

[jgaehring]

Hm, ok, that's discouraging. It's very possible that, as I mentioned above (#324 (comment)), there are 2 separate components of Apple's underlying tech that are breaking the login for us. If that's the case, even if we might have resolved the first one, we still have to fix the second, which means a pretty big change to our login procedure. We'll try to get that rolled out as soon as possible; however, I've been putting off some other improvements for Android users while I deal with these Apple bugbears, so I desperately need to attend to those first. I'll get back to you when we're ready to rollout the fix to login for Apple.

[mstenta]

@ludwa6 - As @jgaehring said it seems like the only way we can solve this issue in the iOS Native app of Field Kit is by implementing OAuth2 login: #311

However, as @jgaehring described in his comment above (#324 (comment)), it may still be possible to change the cookie setting in your iPhone's Safari browser, and then use the app at https://farmOS.app (via your browser, not via the native app from the app store).

I am going to test this approach out with another person who is experiencing the same thing in the iOS native app today. If you can test it out too, that would be great.

I share @jgaehring 's general hesitation in recommending this approach, because it is essentially encouraging you to disable a security setting on your phone. But my thought is: this is a new setting that was not enabled previously, so you are simply reverting that temporarily. And: once we have OAuth2 you can change it back. We will try to remember to remind you and others to do that when the time comes - although it may be hard to communicate that in a consistent way. So I would recommend subscribing to the OAuth2 issue and following for that to be completed as well: #311

[mstenta]

Update: I can confirm that disabling "Prevent cross-site cookies" in Safari on iPhone allows login via https://farmOS.app (but NOT in the native app).

So @ludwa6 - that is something you can do to use Field Kit on your phone for the immediate future, until #311 is done.

[ludwa6]

Yep: thanks @jgaehring and @mstenta : while i look forward to having the native app working, this is good enough for now.

[jgaehring]

Glad to hear that @ludwa6.

I'm going to rename this issue to reflect the core problem (authentication), so that others might find it easier. For the time being, I'll leave it open, since it's not resolved as far as I see it, but as @mstenta pointed out, the main issue we'll use for tracking progress on this bug will be #311.

[jgaehring]

A quick update here. The 0.5.0 release is now live on https://farmos.app (might need to refresh and/or close and reopen the tab for the update to get pulled), and will be rolling out to Android later today. It has OAuth implemented and hopefully resolves a lot of the issues we saw here. As noted here, it requires your server to have farmOS version 1.4 or higher. It does not require that "Prevent cross-site cookies" in Safari be disabled.

The native iOS version 0.5.1, with the same changes plus some iOS specific stuff, should be rolling out sometime this weekend or early next week. 🤞

[jgaehring]

Closing this. Seems to have been remedied with recent updates. @ludwa6, if you get a chance, give the new version a try and let me know how it works.