Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Field Kit Logs With Images Not Uploading to Server #365

Closed
vital-agro opened this issue Jul 30, 2020 · 22 comments
Closed

Field Kit Logs With Images Not Uploading to Server #365

vital-agro opened this issue Jul 30, 2020 · 22 comments

Comments

@vital-agro
Copy link

vital-agro commented Jul 30, 2020 •
edited
Loading

After syncing three test logs, I noticed that only the logs that were created offline were uploaded to our farmOS server. See screenshots attached.

Logs were created using iOS PWA saved to the home screen.FK version 0.5.1 on iOS 13.6 with safari 13.
@vital-agro
Copy link
Author

IMG_1659
IMG_1660

@jgaehring
Copy link
Member

Thanks @vital-agro. Am I correct that you're also getting duplicate logs sent to your server? Are you getting any error messages in Field Kit when you try to sync?

@vital-agro
Copy link
Author

vital-agro commented Jul 30, 2020 via email

@jgaehring
Copy link
Member

Also, I just noticed some bizarre behavior when testing on iPad right now, after coming out of airplane mode, the sync button was non-responsive for a moment, no error messages. This was resolved when I went to the home screen and then reopened the app. Perhaps this might work for you?

I synced all the logs twice and the test cache log was duplicated.

Hm, this makes me think there could be an issue with a database migration, possibly some corrupt data. Were you on a different version before 0.5.1? If so, do you happen to recall the version number?

If you don't have any data you need to save, it might help to logout and login again. That will clear the database and start fresh.

@vital-agro
Copy link
Author

I think I was using 0.5.1. I logged in and out but logs still aren't uploading, so it may be a data migration issue.

Thanks

@vital-agro
Copy link
Author

vital-agro commented Jul 30, 2020 •
edited
Loading

Jamie: I reinstalled the home screen PWA and now the logs are showing up on the server, but after turning cellular service and WiFi off and on, any new logs won't upload.

@jgaehring
Copy link
Member

Hm, I'm having trouble reproducing. The only issue I'm experiencing by coming in and out of airplane mode is the behavior I explained above, where the sync button, and all other buttons, are non-responsive for a time, until I go back to the home screen and reopen the app. Could this be what happens to you?

When you say new logs won't upload, what is the exact behavior? Do the cloud icons for each unsynced log become spinners? For how long do they spin? Or are you seeing no visual response to tapping the sync button? And you're still seeing no error messages?

@vital-agro
Copy link
Author

vital-agro commented Jul 30, 2020 •
edited
Loading

I deleted the PWA from the home screen and then reinstalled it, which enabled me to upload several logs but once I went offline and then online (turning off WiFi and Cellular) I could no longer sync logs. I don't get unresponsive sync icons, and it runs the sync icon for about a second. Still no error messages

I logged out and reinstalled the home screen app again to reproduce the issue and it's definitely the act of turning off cell service or WiFi that triggers the issue.

@vital-agro
Copy link
Author

Update: after several reinstall attempts it seems to be working online and offline for now.

@vital-agro
Copy link
Author

Jamie: I think I found the issue. It only fails to upload when I try to upload an image that was captured via the app while creating the log. I was able to upload previously captured images.

Let me know if you can reproduce that.

@jgaehring
Copy link
Member

Oh that's interesting. Hm, I still can't reproduce. Are there any other specific attributes on the log? What log type are you sending?

@vital-agro
Copy link
Author

It's an activity log with gps coordinates and an image attachment.

@vital-agro
Copy link
Author

Hmm. Now I can't upload logs with any photos.

@jgaehring
Copy link
Member

Hmm, very strange. Do you have many logs with photos in the app? At this point we're not doing much to optimize how they're stored, so there could be a limit to how many you can have stored at one time. I don't know how that would affect uploading, but maybe if you tried deleting some first? Sorry, kinda grasping at straws here, since I still haven't reproduced what you're seeing. 😕

@vital-agro
Copy link
Author

vital-agro commented Jul 30, 2020 •
edited
Loading

I deleted all cached logs on my device and I still can't upload any logs with images, so I don't think that's the issue. I was able to upload more logs without images but they start to duplicate once I try to add logs with images. Also the app tends to be unresponsive at times until I close and reopen it.

@vital-agro vital-agro changed the title Field Kit Logs Not Uploading to FarmOS server Field Kit Logs With Images Not Uploading to Server Jul 30, 2020
@jgaehring
Copy link
Member

I'm starting to wonder if this might be something to do with how your server is configured. @mstenta, do you think there's any reason why restws_file might be failing to accept images?

Also the app tends to be unresponsive at times until I close and reopen it.

Ok, yea, this is what I see from time to time too. Doing a little research now, it seems like this is a known bug: https://bugs.webkit.org/show_bug.cgi?id=211018. Bummer 😕

Btw, thanks for sticking with this, Mason, I really appreciate the feedback. Hopefully we can get this resolved soon!

@vital-agro
Copy link
Author

@paul121 mentioned something about our server configuration in relation to this issue

@mstenta
Copy link
Member

mstenta commented Jul 31, 2020

Ah hmm, nothing comes to mind - but I wonder... @vital-agro it might be worth testing uploading to test.farmos.net, and see if that behaves the same or not. @paul121 can give you credentials to it. If you have the chance, that might be a good way to see if it's a server issue or a Field Kit issue.

@vital-agro
Copy link
Author

Thanks @mstenta, looks like it's a server issue. I was able to upload image logs to the test server.

@mstenta
Copy link
Member

mstenta commented Jul 31, 2020

Good to hear that it was not a Field Kit issue! Happy to help figure out the server issue!

@paul121
Copy link
Member

paul121 commented Jul 31, 2020

I'm able to replicate this using latest FK in Firefox on Android. It's worth noting that the farmOS server is still on v1.4. This should be OK with FK v0.5.1 though?

I just applied this commit to allow OPTIONS to `farm/areas/geojson' thinking it might be related (if that request fails, would the rest of the sync process fail?) but it doesn't seem to have fixed the problem.

What's interesting are the farmOS server logs... when I create a log in FK that has an image, no POST request is ever sent to the server. Logs without images go through fine. See logs below. This seems weird - @mstenta anything server side that might be causing this?

172.18.0.3 - - [31/Jul/2020:15:31:38 +0000] "OPTIONS /taxonomy_term.json?bundle=farm_quantity_units&page=0 HTTP/1.0" 200 582 "https://farmos.app/" "Mozilla/5.0 (Android 10; Mobile; rv:68.0) Gecko/68.0 Firefox/68.0"
172.18.0.3 - - [31/Jul/2020:15:31:38 +0000] "OPTIONS /taxonomy_vocabulary.json HTTP/1.0" 200 582 "https://farmos.app/" "Mozilla/5.0 (Android 10; Mobile; rv:68.0) Gecko/68.0 Firefox/68.0"
172.18.0.3 - - [31/Jul/2020:15:31:38 +0000] "OPTIONS /taxonomy_term.json?bundle=farm_log_categories&page=0 HTTP/1.0" 200 582 "https://farmos.app/" "Mozilla/5.0 (Android 10; Mobile; rv:68.0) Gecko/68.0 Firefox/68.0"
172.18.0.3 - - [31/Jul/2020:15:31:38 +0000] "OPTIONS /log.json?done=0&log_owner=3&page=0 HTTP/1.0" 200 582 "https://farmos.app/" "Mozilla/5.0 (Android 10; Mobile; rv:68.0) Gecko/68.0 Firefox/68.0"
172.18.0.3 - - [31/Jul/2020:15:31:38 +0000] "OPTIONS /farm_asset.json?&archived=0&page=0 HTTP/1.0" 200 582 "https://farmos.app/" "Mozilla/5.0 (Android 10; Mobile; rv:68.0) Gecko/68.0 Firefox/68.0"
172.18.0.3 - - [31/Jul/2020:15:31:38 +0000] "GET /log.json?done=0&log_owner=3&page=0 HTTP/1.0" 200 1023 "https://farmos.app/" "Mozilla/5.0 (Android 10; Mobile; rv:68.0) Gecko/68.0 Firefox/68.0"
172.18.0.3 - - [31/Jul/2020:15:31:38 +0000] "GET /taxonomy_vocabulary.json HTTP/1.0" 200 2270 "https://farmos.app/" "Mozilla/5.0 (Android 10; Mobile; rv:68.0) Gecko/68.0 Firefox/68.0"
172.18.0.3 - - [31/Jul/2020:15:31:38 +0000] "GET /taxonomy_term.json?bundle=farm_quantity_units&page=0 HTTP/1.0" 200 2616 "https://farmos.app/" "Mozilla/5.0 (Android 10; Mobile; rv:68.0) Gecko/68.0 Firefox/68.0"
172.18.0.3 - - [31/Jul/2020:15:31:38 +0000] "GET /taxonomy_term.json?bundle=farm_log_categories&page=0 HTTP/1.0" 200 4322 "https://farmos.app/" "Mozilla/5.0 (Android 10; Mobile; rv:68.0) Gecko/68.0 Firefox/68.0"
172.18.0.3 - - [31/Jul/2020:15:31:39 +0000] "OPTIONS /log HTTP/1.0" 200 582 "https://farmos.app/" "Mozilla/5.0 (Android 10; Mobile; rv:68.0) Gecko/68.0 Firefox/68.0"
172.18.0.3 - - [31/Jul/2020:15:31:39 +0000] "OPTIONS /taxonomy_term.json?vocabulary=3&&page=0 HTTP/1.0" 200 582 "https://farmos.app/" "Mozilla/5.0 (Android 10; Mobile; rv:68.0) Gecko/68.0 Firefox/68.0"
172.18.0.3 - - [31/Jul/2020:15:31:38 +0000] "GET /farm_asset.json?&archived=0&page=0 HTTP/1.0" 200 113806 "https://farmos.app/" "Mozilla/5.0 (Android 10; Mobile; rv:68.0) Gecko/68.0 Firefox/68.0"
172.18.0.3 - - [31/Jul/2020:15:31:40 +0000] "OPTIONS /farm_asset.json?&archived=0&page=1 HTTP/1.0" 200 582 "https://farmos.app/" "Mozilla/5.0 (Android 10; Mobile; rv:68.0) Gecko/68.0 Firefox/68.0"
172.18.0.3 - - [31/Jul/2020:15:31:39 +0000] "GET /taxonomy_term.json?vocabulary=3&&page=0 HTTP/1.0" 200 132359 "https://farmos.app/" "Mozilla/5.0 (Android 10; Mobile; rv:68.0) Gecko/68.0 Firefox/68.0"
172.18.0.3 - - [31/Jul/2020:15:31:40 +0000] "OPTIONS /taxonomy_term.json?vocabulary=3&&page=1 HTTP/1.0" 200 582 "https://farmos.app/" "Mozilla/5.0 (Android 10; Mobile; rv:68.0) Gecko/68.0 Firefox/68.0"
172.18.0.3 - - [31/Jul/2020:15:31:40 +0000] "GET /farm_asset.json?&archived=0&page=1 HTTP/1.0" 200 109243 "https://farmos.app/" "Mozilla/5.0 (Android 10; Mobile; rv:68.0) Gecko/68.0 Firefox/68.0"
172.18.0.3 - - [31/Jul/2020:15:31:42 +0000] "OPTIONS /farm_asset.json?&archived=0&page=2 HTTP/1.0" 200 582 "https://farmos.app/" "Mozilla/5.0 (Android 10; Mobile; rv:68.0) Gecko/68.0 Firefox/68.0"
172.18.0.3 - - [31/Jul/2020:15:31:40 +0000] "GET /taxonomy_term.json?vocabulary=3&&page=1 HTTP/1.0" 200 142382 "https://farmos.app/" "Mozilla/5.0 (Android 10; Mobile; rv:68.0) Gecko/68.0 Firefox/68.0"
172.18.0.3 - - [31/Jul/2020:15:31:42 +0000] "OPTIONS /taxonomy_term.json?vocabulary=3&&page=2 HTTP/1.0" 200 582 "https://farmos.app/" "Mozilla/5.0 (Android 10; Mobile; rv:68.0) Gecko/68.0 Firefox/68.0"
172.18.0.3 - - [31/Jul/2020:15:31:42 +0000] "GET /farm_asset.json?&archived=0&page=2 HTTP/1.0" 200 18210 "https://farmos.app/" "Mozilla/5.0 (Android 10; Mobile; rv:68.0) Gecko/68.0 Firefox/68.0"
172.18.0.3 - - [31/Jul/2020:15:31:42 +0000] "GET /taxonomy_term.json?vocabulary=3&&page=2 HTTP/1.0" 200 51962 "https://farmos.app/" "Mozilla/5.0 (Android 10; Mobile; rv:68.0) Gecko/68.0 Firefox/68.0"

@paul121
Copy link
Member

paul121 commented Jul 31, 2020

Aha! @mstenta diagnosed the problem and prescribed a solution 👏

The problem was file uploads were limited to 1MB and being blocked by the proxy. The images we were testing were > 5MB, but I was able to upload a 500kb image from Field Kit.

To fix I made the following changes on the server:

# Allow large file uploads.
client_max_body_size 50M;

https://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size

Add the following PHP settings:

 'upload_max_filesize=50M';
 'post_max_size=50M';

@jgaehring jgaehring mentioned this issue Jul 31, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@mstenta @paul121 @jgaehring @vital-agro