ODB Regression: OneDrive API query is now returning '500 Internal Server Error' #1035

abraunegg · 2019-04-01T17:09:38Z

Expected or Desired Behavior

Query to return as per normal

Observed Behavior

System generates a 500 Internal Server Error

Example 1:

* Found bundle for host graph.microsoft.com: 0x1779690 [can pipeline]
* Re-using existing connection! (#1) with host graph.microsoft.com
* Connected to graph.microsoft.com (20.190.143.51) port 443 (#1)
> GET /v1.0/drives/b!6N1Mt659TEusd5RKylmza9702P55GblPqKvhABPZsZHDUbDSFm8MR4y7hgfkXWDK/items/01Y3WOZYF6Y2GOVW7725BZO354PWSELRRZ/delta?select=id,name,eTag,cTag,deleted,file,folder,root,fileSystemInfo,remoteItem,parentReference,size HTTP/1.1
Host: graph.microsoft.com
User-Agent: Phobos-std.net.curl/2.085 (libcurl/7.61.1)
Accept: */*

< HTTP/1.1 500 Internal Server Error
< Cache-Control: private
< Transfer-Encoding: chunked
< Content-Type: application/json
< request-id: 8101700f-2429-432e-b89d-de9298dad926
< client-request-id: 8101700f-2429-432e-b89d-de9298dad926
< x-ms-ags-diagnostic: {"ServerInfo":{"DataCenter":"UK South","Slice":"SliceC","Ring":"5","ScaleUnit":"002","RoleInstance":"AGSFE_IN_2","ADSiteName":"UKS"}}
< Duration: 398.7663
< Strict-Transport-Security: max-age=31536000
< Date: Sun, 31 Mar 2019 19:07:36 GMT
< 
OneDrive HTTP Server Response: 500

Example 2:

* Found bundle for host graph.microsoft.com: 0x20db4d0 [can pipeline]
* Re-using existing connection! (#1) with host graph.microsoft.com
* Connected to graph.microsoft.com (20.190.143.50) port 443 (#1)
> GET /v1.0/drives/b!6N1Mt659TEusd5RKylmza9702P55GblPqKvhABPZsZHDUbDSFm8MR4y7hgfkXWDK/items/01Y3WOZYF6Y2GOVW7725BZO354PWSELRRZ/delta?select=id,name,eTag,cTag,deleted,file,folder,root,fileSystemInfo,remoteItem,parentReference,size HTTP/1.1
Host: graph.microsoft.com
User-Agent: Phobos-std.net.curl/2.085 (libcurl/7.59.0)
Accept: */*

< HTTP/1.1 500 Internal Server Error
< Cache-Control: private
< Transfer-Encoding: chunked
< Content-Type: application/json
< request-id: c4540d8a-e406-419a-bfad-232700f6c2b0
< client-request-id: c4540d8a-e406-419a-bfad-232700f6c2b0
< x-ms-ags-diagnostic: {"ServerInfo":{"DataCenter":"UK South","Slice":"SliceC","Ring":"5","ScaleUnit":"001","RoleInstance":"AGSFE_IN_2","ADSiteName":"UKS"}}
< Duration: 399.8826
< Strict-Transport-Security: max-age=31536000
< Date: Sun, 31 Mar 2019 20:26:52 GMT
< 
OneDrive HTTP Server Response: 500
* Connection #1 to host graph.microsoft.com left intact
OneDrive returned a 'HTTP 5xx Server Side Error' - gracefully handling error
OneDrive API Response: {"error":{"code":"generalException","innerError":{"date":"2019-03-31T20:26:52","request-id":"c4540d8a-e406-419a-bfad-232700f6c2b0"},"message":"An unspecified error has occurred."}}

Steps to Reproduce

Perform a query as per above using OneDrive Business Accounts

The text was updated successfully, but these errors were encountered:

ghost · 2019-04-01T17:09:42Z

Thank you for your contribution to OneDrive API Docs. We will be triaging your incoming issue as soon as possible.

ificator · 2019-04-01T20:58:46Z

We tracked down the root cause and believe it has now been resolved - can you confirm?

abraunegg · 2019-04-01T21:28:52Z

@ificator
Can confirm from users that the issue appears to be resolved now. Thanks for the quick turn around.

helderdaniel · 2019-04-01T21:29:26Z

Well, I can tell you that I have experienced this issue the whole day, but in the last 2 hours, at least as I can track, it did not happened again.

abraunegg · 2019-04-02T00:23:06Z

Closing issue as resolved

helderdaniel · 2019-04-02T12:12:39Z

Hi,

this issue just happened again.

The log of the error:

Remaining free space: 1046918357344
[DEBUG] Checking file: ./00-ongoingOne/00-ML-infoBN/artemyk-ibsgd at iclr2018.url
Uploading new file ./00-ongoingOne/00-ML-infoBN/artemyk-ibsgd at iclr2018.url ...
OneDrive returned a 'HTTP 5xx Server Side Error' - gracefully handling error
Uploading 100% |oooooooooooooooooooooooooooooooooooooooo|   ETA   --:--:--:Giving up on sync after three attempts: Key not found: uploadUrl
std.json.JSONException@std/json.d(540): Key not found: uploadUrl
----------------

helderdaniel · 2019-04-02T13:03:17Z

Full error log on the issue that keeps happening:

Remaining free space: 1046821855970
[DEBUG] Checking file: ./00-ongoingOne/NeuralNetworks/Michael_ Nielsen._Neural_Networks_and_Deep_Learning/neural-networks-and-deep-learning-master/fig/weight_initialization_100.json
* Found bundle for host graph.microsoft.com: 0x55dda67da070 [can pipeline]
* Re-using existing connection! (#1) with host graph.microsoft.com
* Connected to graph.microsoft.com (20.190.129.97) port 443 (#1)
> GET /v1.0/me/drive/root:/.%2F00-ongoingOne%2FNeuralNetworks%2FMichael_%20Nielsen._Neural_Networks_and_Deep_Learning%2Fneural-networks-and-deep-learning-master%2Ffig%2Fweight_initialization_100.json:/?select=id,name,eTag,cTag,deleted,file,folder,root,fileSystemInfo,remoteItem,parentReference,size HTTP/1.1
Host: graph.microsoft.com
User-Agent: Phobos-std.net.curl/2.084 (libcurl/7.58.0)
Accept: */*

< HTTP/1.1 404 Not Found
< Cache-Control: private
< Transfer-Encoding: chunked
< Content-Type: application/json
< request-id: 85659020-b1a4-4a50-a7a6-8dd247100e23
< client-request-id: 85659020-b1a4-4a50-a7a6-8dd247100e23
< x-ms-ags-diagnostic: {"ServerInfo":{"DataCenter":"North Europe","Slice":"SliceC","Ring":"2","ScaleUnit":"000","RoleInstance":"AGSFE_IN_50","ADSiteName":"NEU"}}
< Duration: 288.5874
< Strict-Transport-Security: max-age=31536000
< Date: Tue, 02 Apr 2019 12:45:33 GMT
< 
OneDrive HTTP Server Response: 404
* Connection #1 to host graph.microsoft.com left intact
Uploading new file ./00-ongoingOne/NeuralNetworks/Michael_ Nielsen._Neural_Networks_and_Deep_Learning/neural-networks-and-deep-learning-master/fig/weight_initialization_100.json ...
* Found bundle for host graph.microsoft.com: 0x55dda67da070 [can pipeline]
* Re-using existing connection! (#1) with host graph.microsoft.com
* Connected to graph.microsoft.com (20.190.129.97) port 443 (#1)
> POST /v1.0/drives/b!nJQI4UfVkkuT3DscKMYwgTdEq3YfLs5MoqnjdcN889m8trHuJL_7To6pfkM0De48/items/01TUY2TI3CHMFLO7RVVNGISZAA2XSQSGLB:/weight_initialization_100.json:/createUploadSession HTTP/1.1
Host: graph.microsoft.com
User-Agent: Phobos-std.net.curl/2.084 (libcurl/7.58.0)
Accept: */*
Content-Type: application/json
Content-Length: 110

* We are completely uploaded and fine
< HTTP/1.1 504 Gateway Timeout
< Cache-Control: private
< Transfer-Encoding: chunked
< Content-Type: application/json
< request-id: 960f6f5c-b478-4a62-adf6-2bcf09d7c5a0
< client-request-id: 960f6f5c-b478-4a62-adf6-2bcf09d7c5a0
< x-ms-ags-diagnostic: {"ServerInfo":{"DataCenter":"North Europe","Slice":"SliceC","Ring":"2","ScaleUnit":"000","RoleInstance":"AGSFE_IN_50","ADSiteName":"NEU"}}
< Duration: 10008.3608
< Strict-Transport-Security: max-age=31536000
< Date: Tue, 02 Apr 2019 12:45:42 GMT
< 
OneDrive HTTP Server Response: 504
* Connection #1 to host graph.microsoft.com left intact
OneDrive returned a 'HTTP 5xx Server Side Error' - gracefully handling error

                                                                                
Uploading 100% |oooooooooooooooooooooooooooooooooooooooo|   ETA   --:--:--:Giving up on sync after three attempts: Key not found: uploadUrl
std.json.JSONException@std/json.d(540): Key not found: uploadUrl
----------------

ificator · 2019-04-02T17:09:05Z

Is this happening continously? The error suggests that Graph made the request to SharePoint and gave up waiting for a response after 10s. From the SharePoint side the request didn't take anywhere near 10s, and so it could be a transient network hiccup.

helderdaniel · 2019-04-02T18:15:56Z

@ificator

Well, I started uploading about 1.9 GBytes data across 81 000 files, about 7 hours ago.

About 10/15 minutes after started, the client exited with the 5xx error.
Restarting the client, it works for more 10/15 minutes, uploads some more files and exits again with the same error.
I keep this restarting several times, when I noticed the client was down again, for about 5 hours.

Now the client is running for more than 1 hour.
It is still synchronising the data.
I'll let it run for some time or until synchronisation finishes and then I'll report what happened.

helderdaniel · 2019-04-03T06:49:09Z

@ificator
The client is still synchronising with no further issues

helderdaniel · 2019-04-03T10:01:57Z

@ificator
It started again.

Here is the log of errors in the past 1:30 hours or so:
(my current local time is 11:01)

/var/log/onedrive/Helder Daniel.onedrive.log:2019-Apr-03 09:36:15.1279547 OneDrive returned a 'HTTP 5xx Server Side Error' - gracefully handling error
/var/log/onedrive/Helder Daniel.onedrive.log:2019-Apr-03 09:47:02.8001832 OneDrive returned a 'HTTP 5xx Server Side Error' - gracefully handling error
/var/log/onedrive/Helder Daniel.onedrive.log:2019-Apr-03 09:53:18.2194454 OneDrive returned a 'HTTP 5xx Server Side Error' - gracefully handling error
/var/log/onedrive/Helder Daniel.onedrive.log:2019-Apr-03 10:16:13.5691597 OneDrive returned a 'HTTP 5xx Server Side Error' - gracefully handling error
/var/log/onedrive/Helder Daniel.onedrive.log:2019-Apr-03 10:19:22.4974213 OneDrive returned a 'HTTP 5xx Server Side Error' - gracefully handling error
/var/log/onedrive/Helder Daniel.onedrive.log:2019-Apr-03 10:21:05.1112534 OneDrive returned a 'HTTP 5xx Server Side Error' - gracefully handling error
/var/log/onedrive/Helder Daniel.onedrive.log:2019-Apr-03 10:32:37.4803854 OneDrive returned a 'HTTP 5xx Server Side Error' - gracefully handling error
/var/log/onedrive/Helder Daniel.onedrive.log:2019-Apr-03 10:36:03.735002 OneDrive returned a 'HTTP 5xx Server Side Error' - gracefully handling error
/var/log/onedrive/Helder Daniel.onedrive.log:2019-Apr-03 10:54:45.8350916 OneDrive returned a 'HTTP 5xx Server Side Error' - gracefully handling error

abraunegg · 2019-04-03T18:24:40Z

@helderdaniel
It would be worth getting the HTTPS debug logs for these events as well to assist in tracking down what is going wrong in the Microsoft environment.

helderdaniel · 2019-04-03T20:06:31Z

@abraunegg @ificator

Ok,

here are 2 logs with some 5xx errors:

debug_output0.log
debug_output1.log

abraunegg · 2019-04-03T20:33:27Z

some examples from log file:

Uploading new file ./00-ongoingOne/00-ML-infoBN/IDNNs-master/idnns/information/mutual_info_estimation.py ...
* Found bundle for host graph.microsoft.com: 0x55dda67da070 [can pipeline]
* Re-using existing connection! (#1) with host graph.microsoft.com
* Connected to graph.microsoft.com (20.190.129.97) port 443 (#1)
> POST /v1.0/drives/b!nJQI4UfVkkuT3DscKMYwgTdEq3YfLs5MoqnjdcN889m8trHuJL_7To6pfkM0De48/items/01TUY2TI3OKU3TXNZQRFFJ2NARCIQXG4QH:/mutual_info_estimation.py:/createUploadSession HTTP/1.1
Host: graph.microsoft.com
User-Agent: Phobos-std.net.curl/2.084 (libcurl/7.58.0)
Accept: */*
Content-Type: application/json
Content-Length: 110

* We are completely uploaded and fine
< HTTP/1.1 504 Gateway Timeout
< Cache-Control: private
< Transfer-Encoding: chunked
< Content-Type: application/json
< request-id: c591dd40-dc2a-46b8-a62e-c419026b05aa
< client-request-id: c591dd40-dc2a-46b8-a62e-c419026b05aa
< x-ms-ags-diagnostic: {"ServerInfo":{"DataCenter":"North Europe","Slice":"SliceC","Ring":"2","ScaleUnit":"000","RoleInstance":"AGSFE_IN_50","ADSiteName":"NEU"}}
< Duration: 10013.1314
< Strict-Transport-Security: max-age=31536000
< Date: Tue, 02 Apr 2019 12:30:28 GMT
< 
OneDrive HTTP Server Response: 504
* Connection #1 to host graph.microsoft.com left intact
OneDrive returned a 'HTTP 5xx Server Side Error' - gracefully handling error

sample 2:

Uploading new file ./00-ongoingOne/NeuralNetworks/Michael_ Nielsen._Neural_Networks_and_Deep_Learning/neural-networks-and-deep-learning-master/fig/mnist_complete_zero.png ...
* Found bundle for host graph.microsoft.com: 0x55dda67da070 [can pipeline]
* Re-using existing connection! (#1) with host graph.microsoft.com
* Connected to graph.microsoft.com (20.190.129.97) port 443 (#1)
> POST /v1.0/drives/b!nJQI4UfVkkuT3DscKMYwgTdEq3YfLs5MoqnjdcN889m8trHuJL_7To6pfkM0De48/items/01TUY2TI3CHMFLO7RVVNGISZAA2XSQSGLB:/mnist_complete_zero.png:/createUploadSession HTTP/1.1
Host: graph.microsoft.com
User-Agent: Phobos-std.net.curl/2.084 (libcurl/7.58.0)
Accept: */*
Content-Type: application/json
Content-Length: 110

* We are completely uploaded and fine
< HTTP/1.1 504 Gateway Timeout
< Cache-Control: private
< Transfer-Encoding: chunked
< Content-Type: application/json
< request-id: 003bcc13-cdaf-4e43-bda8-7deb26e3adf1
< client-request-id: 003bcc13-cdaf-4e43-bda8-7deb26e3adf1
< x-ms-ags-diagnostic: {"ServerInfo":{"DataCenter":"North Europe","Slice":"SliceC","Ring":"2","ScaleUnit":"000","RoleInstance":"AGSFE_IN_50","ADSiteName":"NEU"}}
< Duration: 10001.6865
< Strict-Transport-Security: max-age=31536000
< Date: Tue, 02 Apr 2019 12:39:47 GMT
< 
OneDrive HTTP Server Response: 504
* Connection #1 to host graph.microsoft.com left intact
OneDrive returned a 'HTTP 5xx Server Side Error' - gracefully handling error

Appears to be again an internal Microsoft Data Centre issue

* Issue #445 and OneDrive/onedrive-api-docs#1035 highlight that the current 5xx error messages are too generic and need to be more specific. This patch makes those error messages more specific to the response code from OneDrive API

abraunegg · 2019-04-03T20:44:21Z

@helderdaniel
abraunegg/onedrive#451 will change the generic logging of 5xx errors to be more prescriptive when logged - thus, will be more clear as to the error encountered. It will not negate the need for the --debug-https data to provide the request-id / client-request-id - but at least this will provide the actual response code and reasoning (500 vs 504 etc)

helderdaniel · 2019-04-03T21:44:03Z

@abraunegg

ok,
thx

* Issue #445 and OneDrive/onedrive-api-docs#1035 highlight that the current 5xx error messages are too generic and need to be more specific. This patch makes those error messages more specific to the response code from OneDrive API

ghost · 2019-04-13T22:00:09Z

This issue has been automatically marked as stale because it has marked as resolved but has not had any activity for 10 days. It will be closed if no further activity occurs within 10 days of this comment. Thank you for your contributions to OneDrive API Docs!

* Issue #445 and OneDrive/onedrive-api-docs#1035 highlight that the current 5xx error messages are too generic and need to be more specific. This patch makes those error messages more specific to the response code from OneDrive API

ghost added the Needs: Triage 🔍 Needs initial triage assessment label Apr 1, 2019

abraunegg mentioned this issue Apr 1, 2019

Issue with syncing OneDrive Business on Fedora 29: "HTTP 5xx Server Side Error" abraunegg/onedrive#445

Closed

ificator self-assigned this Apr 1, 2019

ificator added type:bug and removed Needs: Triage 🔍 Needs initial triage assessment labels Apr 1, 2019

ificator added the status:resolved label Apr 1, 2019

abraunegg closed this as completed Apr 2, 2019

abraunegg reopened this Apr 3, 2019

abraunegg mentioned this issue Apr 3, 2019

Implement more meaningful 5xx error responses (Issue #445) abraunegg/onedrive#451

Merged

ghost added the no-recent-activity label Apr 13, 2019

abraunegg closed this as completed Apr 15, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ODB Regression: OneDrive API query is now returning '500 Internal Server Error' #1035

ODB Regression: OneDrive API query is now returning '500 Internal Server Error' #1035

abraunegg commented Apr 1, 2019

ghost commented Apr 1, 2019

ificator commented Apr 1, 2019

abraunegg commented Apr 1, 2019

helderdaniel commented Apr 1, 2019

abraunegg commented Apr 2, 2019

helderdaniel commented Apr 2, 2019 •

edited

helderdaniel commented Apr 2, 2019 •

edited

ificator commented Apr 2, 2019

helderdaniel commented Apr 2, 2019

helderdaniel commented Apr 3, 2019

helderdaniel commented Apr 3, 2019

abraunegg commented Apr 3, 2019

helderdaniel commented Apr 3, 2019

abraunegg commented Apr 3, 2019

abraunegg commented Apr 3, 2019 •

edited

helderdaniel commented Apr 3, 2019

ghost commented Apr 13, 2019

ODB Regression: OneDrive API query is now returning '500 Internal Server Error' #1035

ODB Regression: OneDrive API query is now returning '500 Internal Server Error' #1035

Comments

abraunegg commented Apr 1, 2019

Category

Expected or Desired Behavior

Observed Behavior

Steps to Reproduce

ghost commented Apr 1, 2019

ificator commented Apr 1, 2019

abraunegg commented Apr 1, 2019

helderdaniel commented Apr 1, 2019

abraunegg commented Apr 2, 2019

helderdaniel commented Apr 2, 2019 • edited

helderdaniel commented Apr 2, 2019 • edited

ificator commented Apr 2, 2019

helderdaniel commented Apr 2, 2019

helderdaniel commented Apr 3, 2019

helderdaniel commented Apr 3, 2019

abraunegg commented Apr 3, 2019

helderdaniel commented Apr 3, 2019

abraunegg commented Apr 3, 2019

abraunegg commented Apr 3, 2019 • edited

helderdaniel commented Apr 3, 2019

ghost commented Apr 13, 2019

helderdaniel commented Apr 2, 2019 •

edited

helderdaniel commented Apr 2, 2019 •

edited

abraunegg commented Apr 3, 2019 •

edited