Kinto Error HTTP 507 Insufficient Storage #791

sebastienbarbier · 2018-03-22T04:38:13Z

On authentication using FxA, Kinto print in console an error 507

Error: HTTP 507; Error: HTTP 507 Insufficient Storage: Resource access is forbidden for this user (Collection maximum size exceeded (205369 > 204800 Bytes).)
Stack trace:
formatResponse@moz-extension://6aefee5b-5199-4743-b265-b47b7d8779ce/vendor/kinto.js:4145:21
_callee$@moz-extension://6aefee5b-5199-4743-b265-b47b7d8779ce/vendor/kinto.js:4097:50
r@moz-extension://6aefee5b-5199-4743-b265-b47b7d8779ce/vendor/kinto-http.js:7:204
[2]</</u/<@moz-extension://6aefee5b-5199-4743-b265-b47b7d8779ce/vendor/kinto-http.js:7:1390
[2]</</a/</t[e]@moz-extension://6aefee5b-5199-4743-b265-b47b7d8779ce/vendor/kinto-http.js:7:380
step@moz-extension://6aefee5b-5199-4743-b265-b47b7d8779ce/vendor/kinto.js:3958:183
[9]</_asyncToGenerator/</</step/<@moz-extension://6aefee5b-5199-4743-b265-b47b7d8779ce/vendor/kinto.js:3958:361
s@moz-extension://6aefee5b-5199-4743-b265-b47b7d8779ce/vendor/kinto-http.js:596:1731
[200]</notify/<@moz-extension://6aefee5b-5199-4743-b265-b47b7d8779ce/vendor/kinto-http.js:596:1870
s@moz-extension://6aefee5b-5199-4743-b265-b47b7d8779ce/vendor/kinto-http.js:193:343
  sync.js:265:7

Having lots of async multi-thread stuff going on, I had a couple of infinite loop populating my storage. On sync I receive 370 skipped entries having deleted: true.

Still trying to use it, seams like it keeps increasing with time.
Error: HTTP 507; Error: HTTP 507 Insufficient Storage: Resource access is forbidden for this user (Collection maximum size exceeded (212530 > 204800 Bytes).)

The text was updated successfully, but these errors were encountered:

vladikoff · 2018-03-22T04:40:21Z

@glasserc any thoughts on this?

vladikoff · 2018-03-22T04:40:28Z

cc @Natim

vladikoff · 2018-03-22T04:46:33Z

also cc @bqbn

sebastienbarbier · 2018-03-22T05:05:40Z

My account no longer sync, but notes should have a notification to warn me about this. @ryanfeeley should we design something ?

vladikoff · 2018-03-22T14:36:13Z

from our end:

see if we can delete revisions
handle error
determine the max bytes and show user a warnings

Natim · 2018-03-22T14:39:03Z

Remove quota 👍

vladikoff · 2018-03-22T14:39:37Z

@Natim We are worried that that kinto saves too many old revisions, can we delete old stuff or is that auto?

Natim · 2018-03-22T14:40:12Z

There are no such things as revisions, it always override with the last update. With multiple notes you should ask definitely ask for more storage.

glasserc · 2018-03-22T14:43:01Z

I wouldn't be surprised if there's an issue with quotas on the server side. See also https://bugzilla.mozilla.org/show_bug.cgi?id=1422673. I did in fact ask Wei to run some queries and got some scary numbers back, but I had been putting off actually looking at this until I had a user who could reliably reproduce it so I could try to dig in a bit more. I'll try to write up some debugging instructions later today.

vladikoff · 2018-03-22T14:56:14Z

Thank you @Natim @glasserc !!

glasserc · 2018-03-22T18:18:06Z

So, I just reread the original issue. I thought at first that the issue was that the quota numbers were wrong and that you actually hadn't exceeded the quota. But now I think the problem may be that you actually did exceed the quota and don't know how to un-exceed it?

Here are the debugging instructions I wrote in case you are trying to figure out why the quota numbers are wrong. If they aren't actually wrong, then maybe I misunderstood the issue, and please feel free to add an explanation.

Debugging quota numbers

You can't directly inspect quota records via the HTTP interface, but you can at least check what's being stored on the server. For this you'll need an FXA token. With the fxa_client library, you can do

fxa-client --bearer --auth "my-fxa-email@example.com"
        --account-server https://api.accounts.firefox.com/v1
        --oauth-server https://oauth.accounts.firefox.com/v1
        --scopes "https://identity.mozilla.com/apps/notes" --out env.sh

@vladikoff -- can you confirm for me that the scope is correct? This will write a file called env.sh that will have an environment variable called OAUTH_BEARER_TOKEN in it.

Then, you can make requests using that token.

http GET 'https://webextensions.settings.services.mozilla.com/v1/buckets/default Authorization:"Bearer $OAUTH_BEARER_TOKEN"
http GET 'https://webextensions.settings.services.mozilla.com/v1/buckets/default/collections/notes/records Authorization:"Bearer $OAUTH_BEARER_TOKEN"

I'd be curious to know what your bucket ID is. If you can get that, you can probably ask one of the operations guys (for instance @bqbn) to run SQL queries against the kintotp stack. In particular, assuming your bucket_id is some_bucket_id, you might be curious about the outcome of queries such as

SELECT * FROM records WHERE parent_id LIKE '/buckets/some_bucket_id%' AND collection_id = 'quota';

You might compare these numbers against what is visible when you just request records in the default bucket in the notes collection. (Are there lots of records? You may need to know how to paginate in Kinto.) The sizes should match the string length of the JSON of all the records.

Natim · 2018-03-22T18:27:04Z

To unexeed it you need to remove the object. But I think the issue is that notes need to store more than what is currently authorized so we should ask OPS to actually update quotas to a much higher value.

If you think that a note can be up to 4kb and that we want use to be able to store 100 notes we should make sure they can store up to 500kb

vladikoff · 2018-03-23T15:34:37Z

Hit the limit while testing with @ryanfeeley , we added a 160kb note to an existing list of notes:

vladikoff · 2018-03-23T15:35:04Z

vladikoff · 2018-03-23T15:36:47Z

After I hit the storage limit, and reload the sidebar I get booted out of sync

Natim · 2018-03-23T15:42:30Z

I guess we can raise the limit for notes by a lot, even 1MB or 10MB is fine we only have a couple hundreds thousands of users and we can alocate 100GB for sure 💃

vladikoff · 2018-03-23T15:42:35Z

@sebastienbarbier @glasserc @Natim Even after deleting several large Notes in testing we were not able to get our storage back. So we might need to investigate to make sure deletion actually frees up storage.

Natim · 2018-03-23T15:43:07Z

If you delete singleNotes you should recover it

vladikoff · 2018-03-23T15:54:00Z

after deleting several large Notes in testing we were not able to get our storage back

Ah I see, after we hit a 507 we "lose sync" so deleting anything won't make any network requests

sebastienbarbier · 2018-03-23T16:10:22Z

All right looks like some deep investigation is needed!
@ryanfeeley should be have some sort of notification ?! May be limiting size or number of notes ?! 🤔

vladikoff · 2018-03-27T23:50:44Z

@Natim is there a way to tell how much space is remaining in Kinto (for users's notes) OR do we have to calculate that ourselves?

vladikoff · 2018-03-28T03:23:34Z

Filed a bug on the server to figure out server limits: https://bugzilla.mozilla.org/show_bug.cgi?id=1449402 but we still need to handle errors as part of this issue

vladikoff · 2018-03-28T03:59:38Z

Natim · 2018-03-28T08:26:44Z

According to https://testpilot.settings.services.mozilla.com/v1/ we don't show it here. We could/should.

After authenticating, kinto might return some errors we should handle. fix synced UI after kinto error. Should request reconnect. implement a solution to unblock the user. display used quota. #837 Fix #791 #801 #802 #827

wellington1993 · 2018-09-03T22:44:47Z

Hi,

I'm sorry for reopen, but I've got this error today:

1536007467493	Sync.ErrorHandler	DEBUG	extension-storage failed: Error: HTTP 507 Insufficient Storage: Resource access is forbidden for this user (Maximum bytes per object exceeded (22031 > 16384 Bytes.)(resource://services-common/kinto-http-client.js:2341:5) JS Stack trace: ServerResponse@kinto-http-client.js:2341:5
processResponse@kinto-http-client.js:2469:13
async*request@kinto-http-client.js:2521:14
async*execute@kinto-http-client.js:769:26
async*_batchRequests@kinto-http-client.js:685:33
async*batch@kinto-http-client.js:732:29
async*wrappedMethod@kinto-http-client.js:2931:18
batch@kinto-http-client.js:2191:12
async*pushChanges@kinto-offline-client.js:1805:26
async*sync@kinto-offline-client.js:1974:13
async*_syncCollection/<@ExtensionStorageSync.jsm:814:14
_requestWithToken@ExtensionStorageSync.jsm:825:20
async*_syncCollection@ExtensionStorageSync.jsm:806:12
sync@ExtensionStorageSync.jsm:744:27
async*syncAll/promises</<@ExtensionStorageSync.jsm:718:16
promise callback*syncAll/promises<@ExtensionStorageSync.jsm:717:63
syncAll@ExtensionStorageSync.jsm:716:22
async*_sync@extension-storage.js:38:12
async*WrappedNotify@util.js:179:27
async*sync@engines.js:895:12
async*_syncEngine@enginesync.js:214:13
async*sync@enginesync.js:156:21
async*onNotify@service.js:1139:13
async*WrappedNotify@util.js:179:27
async*WrappedLock@util.js:135:22
async*_lockedSync@service.js:1132:12
async*sync/<@service.js:1124:13
async*WrappedCatch@util.js:105:22
async*sync@service.js:1113:12
async*syncIfMPUnlocked/<@policies.js:537:7

There is a relation with this issue and that error?

Thanks.
error-sync-1536007467508.txt
error-sync-1536013518287.txt
error-sync-1536014153919.txt

glasserc · 2018-09-18T19:00:32Z

No, probably not, because Notes doesn't use extension-storage as far as I know.

zacc1111 · 2019-10-13T02:19:54Z

... i have the same Problem.

1570931509947 Sync.Engine.Extension-Storage INFO Successfully synced 'jid1-vs5odTmtIydjMg@jetpack'
1570931510282 Sync.Engine.Extension-Storage ERROR Syncing {73a6fe31-595d-460b-a920-fcc0f8843232}: request failed: Error: HTTP 507 Insufficient Storage: Resource access is forbidden for this user (Maximum bytes per object exceeded " "(17467 > 16384 Bytes.)(resource://services-common/kinto-http-client.js:2771:5) JS Stack trace: ServerResponse@kinto-http-client.js:2771:5
processResponse@kinto-http-client.js:2918:13
1570931510282 Sync.Engine.Extension-Storage WARN Syncing failed: Error: HTTP 507 Insufficient Storage: Resource access is forbidden for this user (Maximum bytes per object exceeded " "(17467 > 16384 Bytes.)(resource://services-common/kinto-http-client.js:2771:5) JS Stack trace: ServerResponse@kinto-http-client.js:2771:5
processResponse@kinto-http-client.js:2918:13
1570931510282 Sync.Status DEBUG Status for engine extension-storage: error.engine.reason.unknown_fail
1570931510282 Sync.Status DEBUG Status.service: success.status_ok => error.sync.failed_partial
1570931510282 Sync.ErrorHandler DEBUG extension-storage failed: Error: HTTP 507 Insufficient Storage: Resource access is forbidden for this user (Maximum bytes per object exceeded " "(17467 > 16384 Bytes.)(resource://services-common/kinto-http-client.js:2771:5) JS Stack trace: ServerResponse@kinto-http-client.js:2771:5
processResponse@kinto-http-client.js:2918:13
1570931510283 Sync.Doctor INFO Skipping check of prefs - disabled via preferences
1570931510283 Sync.Doctor INFO Skipping check of passwords - disabled via preferences
1570931510283 Sync.Doctor INFO Skipping check of tabs - disabled via preferences
1570931510283 Sync.Doctor INFO Skipping check of bookmarks - disabled via preferences
1570931510283 Sync.Doctor INFO Skipping check of addons - disabled via preferences
1570931510283 Sync.Doctor INFO Skipping check of forms - disabled via preferences
1570931510283 Sync.Doctor INFO Skipping check of history - disabled via preferences
1570931510283 Sync.Doctor INFO Skipping check of extension-storage - disabled via preferences
1570931510283 Sync.Doctor INFO Skipping validation: no engines qualify
1570931510283 Sync.Synchronizer INFO Sync completed at 2019-10-13 03:51:50 after 3.50 secs.
1570931510284 Sync.Declined DEBUG Handling remote declined: []
1570931510284 Sync.Declined DEBUG Handling local declined: []
1570931510284 Sync.Declined DEBUG Declined changed? false
1570931510284 Sync.Service INFO No change to declined engines. Not reuploading meta/global.
1570931510284 Sync.ErrorHandler ERROR Some engines did not sync correctly.
1570931510284 Sync.SyncScheduler DEBUG Next sync in 3600000 ms. (why=schedule)

vladikoff added this to the Triaged milestone Mar 22, 2018

vladikoff assigned sebastienbarbier Mar 23, 2018

vladikoff mentioned this issue Mar 26, 2018

You are wrongly logged out from "Firefox Notes" after reopening the sidebar if a sync action was previously performed #802

Closed

sebastienbarbier mentioned this issue Mar 28, 2018

Handle sync errors #824

Merged

4 tasks

vladikoff closed this as completed in #824 Mar 29, 2018

casual-will mentioned this issue Apr 26, 2020

Sync Containers with Temporary Account Containers mozilla/multi-account-containers#1675

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Kinto Error HTTP 507 Insufficient Storage #791

Kinto Error HTTP 507 Insufficient Storage #791

sebastienbarbier commented Mar 22, 2018 •

edited

Loading

vladikoff commented Mar 22, 2018

vladikoff commented Mar 22, 2018

vladikoff commented Mar 22, 2018

sebastienbarbier commented Mar 22, 2018

vladikoff commented Mar 22, 2018

Natim commented Mar 22, 2018

vladikoff commented Mar 22, 2018

Natim commented Mar 22, 2018 •

edited

Loading

glasserc commented Mar 22, 2018

vladikoff commented Mar 22, 2018

glasserc commented Mar 22, 2018 •

edited

Loading

Natim commented Mar 22, 2018

vladikoff commented Mar 23, 2018

vladikoff commented Mar 23, 2018

vladikoff commented Mar 23, 2018

Natim commented Mar 23, 2018 •

edited

Loading

vladikoff commented Mar 23, 2018

Natim commented Mar 23, 2018

vladikoff commented Mar 23, 2018

sebastienbarbier commented Mar 23, 2018

vladikoff commented Mar 27, 2018

vladikoff commented Mar 28, 2018

vladikoff commented Mar 28, 2018

Natim commented Mar 28, 2018

wellington1993 commented Sep 3, 2018

glasserc commented Sep 18, 2018

zacc1111 commented Oct 13, 2019

Kinto Error HTTP 507 Insufficient Storage #791

Kinto Error HTTP 507 Insufficient Storage #791

Comments

sebastienbarbier commented Mar 22, 2018 • edited Loading

vladikoff commented Mar 22, 2018

vladikoff commented Mar 22, 2018

vladikoff commented Mar 22, 2018

sebastienbarbier commented Mar 22, 2018

vladikoff commented Mar 22, 2018

Natim commented Mar 22, 2018

vladikoff commented Mar 22, 2018

Natim commented Mar 22, 2018 • edited Loading

glasserc commented Mar 22, 2018

vladikoff commented Mar 22, 2018

glasserc commented Mar 22, 2018 • edited Loading

Debugging quota numbers

Natim commented Mar 22, 2018

vladikoff commented Mar 23, 2018

vladikoff commented Mar 23, 2018

vladikoff commented Mar 23, 2018

Natim commented Mar 23, 2018 • edited Loading

vladikoff commented Mar 23, 2018

Natim commented Mar 23, 2018

vladikoff commented Mar 23, 2018

sebastienbarbier commented Mar 23, 2018

vladikoff commented Mar 27, 2018

vladikoff commented Mar 28, 2018

vladikoff commented Mar 28, 2018

Natim commented Mar 28, 2018

wellington1993 commented Sep 3, 2018

glasserc commented Sep 18, 2018

zacc1111 commented Oct 13, 2019

sebastienbarbier commented Mar 22, 2018 •

edited

Loading

Natim commented Mar 22, 2018 •

edited

Loading

glasserc commented Mar 22, 2018 •

edited

Loading

Natim commented Mar 23, 2018 •

edited

Loading