Key not found: driveType

[dawbarton]

After updating to the latest master this morning I keep getting the error

Initializing the Synchronization Engine ...
std.json.JSONException@std/json.d(494): Key not found: driveType
----------------
/usr/include/dmd/phobos/std/exception.d:515 pure @safe void std.exception.bailOut!(std.json.JSONException).bailOut(immutable(char)[], ulong, const(char[])) [0x58d593]
??:? pure @safe inout(std.json.JSONValue)* std.exception.enforce!(std.json.JSONException).enforce!(inout(std.json.JSONValue)*).enforce(inout(std.json.JSONValue)*, lazy const(char)[], immutable(char)[], ulong) [0x637c12]
??:? inout pure ref @safe inout(std.json.JSONValue) std.json.JSONValue.opIndex(immutable(char)[]) [0x5e8705]
src/sync.d:165 void sync.SyncEngine.init() [0x5b5972]
src/main.d:229 _Dmain [0x59aa08]

After applying the --debug-https option, it becomes a bit more clear -

< HTTP/1.1 504 Gateway Timeout
< Cache-Control: private
< Transfer-Encoding: chunked
< Content-Type: application/json
< request-id: something
< client-request-id: someotherthing
< x-ms-ags-diagnostic: {"ServerInfo":{"DataCenter":"UK South","Slice":"SliceC","Ring":"5","ScaleUnit":"001","Host":"AGSFE_IN_7","ADSiteName":"UKS"}}
< Duration: 20048.989
< Strict-Transport-Security: max-age=31536000
< Date: Wed, 29 Aug 2018 09:48:15 GMT
< 
* Connection #1 to host graph.microsoft.com left intact
std.json.JSONException@std/json.d(494): Key not found: driveType
----------------
/usr/include/dmd/phobos/std/exception.d:515 pure @safe void std.exception.bailOut!(std.json.JSONException).bailOut(immutable(char)[], ulong, const(char[])) [0x58d593]
??:? pure @safe inout(std.json.JSONValue)* std.exception.enforce!(std.json.JSONException).enforce!(inout(std.json.JSONValue)*).enforce(inout(std.json.JSONValue)*, lazy const(char)[], immutable(char)[], ulong) [0x637c12]
??:? inout pure ref @safe inout(std.json.JSONValue) std.json.JSONValue.opIndex(immutable(char)[]) [0x5e8705]
src/sync.d:165 void sync.SyncEngine.init() [0x5b5972]
src/main.d:229 _Dmain [0x59aa08]

so it looks like the connection timing out is causing some other random error in onedrive. I'm just going to wait and hope that this is just an issue at Microsoft's end. Thanks.

[abraunegg]

There also seems to be some API issues atm - dealing with shared folders.

To rule this out can you try connecting from a more stable connection and see if the issue persists?

[dawbarton]

I'm pretty sure it is just at Microsoft's end as it appears to be working now (my current connection is about as stable as is possible). It might be good to replace the error message with something more explanatory though.

[abraunegg]

Handling random errors is something certainly that needs to be done better in the application - will see what can be done in this instamce

[borouhin]

I've just encountered a similar error, maybe the same cause.
Onedrive ran in monitor mode. A newly added local directory with a lot of small files was uploaded. One file (nothing strange about it) gave an error:

2018-Aug-30 14:05:08.1563025 Uploading file [path skipped] ... done.
2018-Aug-30 14:05:08.2797757 Key not found: size

After that, further uploading stopped, a new remote file was downloaded and the program became idle:

2018-Aug-30 14:05:08.2799043 Syncing changes from OneDrive ...
2018-Aug-30 14:05:34.1144266 Downloading file [path skipped] ... done.

I had to stop onedrive and run it again with "--synchronize --local-first" switches to resume upload (now I have to wait several hours for the program to rescan all the local files logging that they "have not changed" :( )
Unfortunately, I cannot reproduce the bug with verbose logging on.

Generally speaking, I think the logic of the program is problematic in general. Every minor error with a single file may lead to termination of the whole sync task or even to crash. Such errors are inevitable when there are a lot of data to sync, and the program should be tolerant to them, so that constant manual intervention in syncing process shouldn't be necessary.
Currently I'm in the process of transferring my 700 Gb file archive to OneDrive - and I doubt it will ever complete successfully, as I encounter one issue after another, each of them forcing me to resync, and each resync taking ages...

[borouhin]

Once again, now with verbose logging on:

Uploading file [path_skipped] ...OneDrive returned a 'HTTP 5xx Server Side Error' - gracefully handling error done.
Key not found: size
Syncing changes from OneDrive ...

The error was indeed handled gracefully, i.e. without crash. But further uploads were not done.

[abraunegg]

So all objects should have a 'size' property as it is being requested. If a 'size' is not present, then there is something going on with the OneDrive API being inconsistent in the details being presented back. It is easy enough to fix the code that in the event that no size property is provided, to assume 0. The impact here is that the 'size' property is being used to determine if there is enough space to upload new files.

[borouhin]

I think there is nothing wrong with the size, it's just a network connectivity issue. That's why I suppose it's the same issue as @dawbarton reported. I think in case of any such unclassified problems the program should just try one more time and if the problem persists - skip to the next item. Otherwise, it's almost impossible to sync large amounts of data with imperfect Internet connection.

[abraunegg]

This is what I have been trying to do - as previously, the application would segfault and exit - mostly all these things are now taken care of - it's the loose ends like this issue that still need closing out.

[abraunegg]

@borouhin, @dawbarton
Can you help test the following PR to resolve these issues identified:

• Unable to correctly initialise due to receiving a 504 (gateway timeout)
• Unable to upload additional files due to an upload getting a 504 (gateway timeout) & exiting upload process

git clone https://github.com/abraunegg/onedrive.git
cd onedrive
git fetch origin pull/166/head:pr166
git checkout pr166
make
make install

[abraunegg]

@borouhin, @dawbarton
Any update on being able to test this PR?

[dawbarton]

Will try to check this evening when I'm on my home internet connection (which is more flaky than my work one).

[borouhin]

I'm testing the new branch now, will resopond ASAP.

[borouhin]

Unfortunately, the PR didn't help. the program crashed after logging an attempt to handle error gracefully:

Uploading file [path skipped]... done.
Remaining free space: 602516197029
OneDrive returned a 'HTTP 5xx Server Side Error' - gracefully handling error
std.json.JSONException@std/json.d(494): Key not found: name
----------------
/home/alb/dlang/dmd-2.081.2/linux/bin64/../../src/phobos/std/exception.d:515 pure @safe void std.exception.bailOut!(std.json.JSONException).bailOut(immutable(char)[], ulong, scope const(char)[]) [0x14c42797]
??:? pure @safe inout(std.json.JSONValue)* std.exception.enforce!(std.json.JSONException).enforce!(inout(std.json.JSONValue)*).enforce(inout(std.json.JSONValue)*, lazy const(char)[], immutable(char)[], ulong) [0x14ce6296]
??:? inout pure ref @safe inout(std.json.JSONValue) std.json.JSONValue.opIndex(immutable(char)[]) [0x14c999b9]
src/sync.d:1307 void sync.SyncEngine.uploadNewFile(immutable(char)[]) [0x14c6fc91]
src/sync.d:1059 void sync.SyncEngine.uploadNewItems(immutable(char)[]) [0x14c6eab9]
src/sync.d:1050 void sync.SyncEngine.uploadNewItems(immutable(char)[]) [0x14c6e9b0]
src/sync.d:1050 void sync.SyncEngine.uploadNewItems(immutable(char)[]) [0x14c6e9b0]
src/sync.d:1050 void sync.SyncEngine.uploadNewItems(immutable(char)[]) [0x14c6e9b0]
src/sync.d:1050 void sync.SyncEngine.uploadNewItems(immutable(char)[]) [0x14c6e9b0]
src/sync.d:1050 void sync.SyncEngine.uploadNewItems(immutable(char)[]) [0x14c6e9b0]
src/sync.d:1050 void sync.SyncEngine.uploadNewItems(immutable(char)[]) [0x14c6e9b0]
src/sync.d:777 void sync.SyncEngine.scanForDifferences(immutable(char)[]) [0x14c6d691]
src/main.d:430 void main.performSync(sync.SyncEngine, immutable(char)[], bool, bool, bool) [0x14c507a3]
src/main.d:294 _Dmain [0x14c50013]

[abraunegg]

@borouhin
Your internet connection reliability reminds me of ADSL days here in AU !!

OK - thats a crash in a different area, updated the PR. If you can retry testing the PR with the latest commits that would be great.

[borouhin]

@abraunegg, optical fiber is hanging on my fence for 2 months already, waiting for being connected. Something not ready on ISP side yet... So currently it's a dedicated WiFi link, theoretically up to 30 Mpbs, but usually not more than 15 Mbps and, most importantly, with a noticeable packet loss.

As for the updated PR. I've tested it and I encountered a similar error. Not "key not found", but "HTTP request returned status code 401 (Unauthorized)" I don't know if it's also because of the bad connection or it's some bug on OneDrive side. But previously such errors caused crashes too.
Now it didn't crash, but started uploading from the very beginning, re-checking all the previously uploaded files and folders (I ran onedrive with --local-first and --single-directory switches, as it's the only way not to waste many hours after each error or crash):

2018-Sep-17 11:14:27.9383477 Uploading file [skipped]... done.
2018-Sep-17 11:14:30.040788 Remaining free space: 556620556867
2018-Sep-17 11:14:44.3134319 HTTP request returned status code 401 (Unauthorized)
{
    "error": {
        "code": "unauthenticated",
        "innerError": {
            "date": "2018-09-17T08:14:44",
            "request-id": "19e5d7ea-1672-4bcf-bb8c-cf43c362008d"
        },
        "message": "Access Denied"
    }
}
2018-Sep-17 11:14:44.3135594 Syncing changes from this selected path: [skipped]
2018-Sep-17 11:14:44.3136277 Syncing changes from selected local path first before downloading changes from OneDrive ...
2018-Sep-17 11:14:44.314166 Uploading differences of [skipped]
2018-Sep-17 11:14:44.3144254 Processing [skipped]
2018-Sep-17 11:14:44.4186397 The directory has not changed

That's better, but still not quite the desired behavior, as re-checking a lot of files takes a lot of time. Ideally after any error with a single file the program should retry with that file and/or proceed to the next one in queue.

[abraunegg]

PR Merged

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.