Sync is bogus with the new version of uBlock

[sander85]

Describe the issue

After migrating to web extension of the uBlock I can't sync my devices anymore. It seems to be able to sync (download) only when I have uploaded something but this would erase all the changes that I have uploaded from the other device.

One or more specific URLs where the issue occurs

moz-extension://cad1c7f1-1273-499c-a9a6-625f965628d9/dashboard.html#dyna-rules.html

Screenshot in which the issue can be seen

Steps for anyone to reproduce the issue

1. Migrate to new extension.
2. Try to pull some new changes from Sync.
3. Only upload button is shown.

Your settings

Sync is enabled, not sure if it's enabled by default.

• OS/version: Linux (Mageia 6)
• Browser/version: Firefox 55.0.3 (64-bit)
• uBlock Origin version: v1.14.4

Your filter lists

I don't think these are relevant in this bug.

Your custom filters (if any)

Same as above.

[gorhill]

I don't understand your issue. In your steps-to-reproduce, there is no step which says "Click 'Export to cloud storage'", hence of course there won't be anything to import from cloud storage.

If your issue is that your cloud storage was filled with legacy uBO data, then it's expected you can't import it with webext version, the Sync data is stored in a different way with webext version. If you are unhappy with this, you will have to close the issue here and open one at bugzilla.

[sander85]

The problem is that I exported it to cloud from another device but am unable to import on other devices. This worked with the legacy version of uBlock.

[gorhill]

I exported it to cloud from another device

You exported with what version of uBO? What Firefox version was it?

I did try the Sync feature, and it worked fine here.

[sander85]

uBO version and Firefox are the same on both systems. But I found this from about:sync-log:

1505156092353	Sync.Engine.Extension-Storage	ERROR	Syncing uBlock0@raymondhill.net: request failed: Error: HTTP 507; Error: HTTP 507 Insufficient Storage: Resource access is forbidden for this user (Maximum bytes per object exceeded (8763 > 8192 Bytes.) (resource://services-common/kinto-http-client.js:2353:21) JS Stack trace: formatResponse@kinto-http-client.js:2376:21 < processResponse@kinto-http-client.js:2351:14 < waitForSyncCallback@async.js:98:7 < makeSpinningCallback/callback.wait@async.js:168:27 < promiseSpinningly@async.js:234:12 < _sync@extension-storage.js:41:12 < WrappedNotify@util.js:160:21 < sync@engines.js:723:5 < _syncEngine@enginesync.js:219:7 < sync@enginesync.js:166:15 < onNotify@service.js:1081:7 < WrappedNotify@util.js:160:21 < WrappedLock@util.js:116:16 < _lockedSync@service.js:1071:12 < sync/<@service.js:1063:7 < WrappedCatch@util.js:91:16 < sync@service.js:1052:5
1505156092354	Sync.Engine.Extension-Storage	WARN	Syncing failed: Error: HTTP 507; Error: HTTP 507 Insufficient Storage: Resource access is forbidden for this user (Maximum bytes per object exceeded (8763 > 8192 Bytes.) (resource://services-common/kinto-http-client.js:2353:21) JS Stack trace: formatResponse@kinto-http-client.js:2376:21 < processResponse@kinto-http-client.js:2351:14 < waitForSyncCallback@async.js:98:7 < makeSpinningCallback/callback.wait@async.js:168:27 < promiseSpinningly@async.js:234:12 < _sync@extension-storage.js:41:12 < WrappedNotify@util.js:160:21 < sync@engines.js:723:5 < _syncEngine@enginesync.js:219:7 < sync@enginesync.js:166:15 < onNotify@service.js:1081:7 < WrappedNotify@util.js:160:21 < WrappedLock@util.js:116:16 < _lockedSync@service.js:1071:12 < sync/<@service.js:1063:7 < WrappedCatch@util.js:91:16 < sync@service.js:1052:5

Not sure if it's Firefox/Sync to blame or if uBO should send smaller objects?

[gorhill]

I internally work with chunks of max size 6,144 character-long string segments, which to me seems safe enough taking into account that this is put through JSON.stringify. What sort of string derived from a ruleset would cause a 6,144 character-long string to expand to 8,763 bytes once JSON.stringify'ied?

[sander85]

This is the list that I have: https://sander85.eu/paste/view/9d08bf45

I don't have any other custom rules.

[gorhill]

Ok, so FF encrypts and then encodes the result as base64. Unfortunately, from what I see the per-item data size limit applies after the encrypting/encoding, so this makes it more difficult to find a safe max chunk size -- both the encryption and encoding add their own data size overhead, and add to this FF's own information (like last_time_modified) is added to the original data. It could be argued with Firefox people that the limit should apply before the encrypting/encoding.

[gorhill]

I just lowered the max chunk size for now to at least avoid failure, I verified it works with your ruleset.

[gorhill]

If you are willing to use the dev build, fix is in 1.14.9b1 in AMO's dev channel.

[sander85]

Thanks a lot! Will test ASAP :)

[sander85]

Weird, I'm still seeing this error:
1505327718411 Sync.ErrorHandler DEBUG extension-storage failed: Error: HTTP 507; Error: HTTP 507 Insufficient Storage: Resource access is forbidden for this user (Maximum bytes per object exceeded (8763 > 8192 Bytes.)

This is with 1.14.9b1

[gorhill]

I just tested again and it worked on my side:

On the left is Firefox 55 with a new profile + uBO dev build + all your rules. On the right is Firefox Nightly, in which you can see I was able to see the rules from the cloud storage, which I successfully imported. Here are the steps I went through:

1. Log in to Sync in both FF 55 and FF Nightly.
2. In FF 55, renamed the device "New Temp Profile" in uBO
3. In FF 55, sent the rules to the cloud
4. In FF 55, clicked the sync button to force a sync
5. In FF Nightly, clicked the sync button to force a sync
6. In FF Nightly, I forced a refresh of the "My rules" pane to force uBO to read whatever data is available in cloud storage
7. Result: FF Nightly showed the data that was sent from FF 55
8. I could then import all the rules, as seen in the screenshot

[sander85]

It seems I had to push the button to upload the rules again and after that it now works. Superb job!

[collinbarrett]

Hmm, I am seeing the same issue right now. I am using Firefox Developer v57.03b and Firefox 55.0.3. I installed the latest dev build 1.14.11rc3 of uBlock Origin.

Sync works for "My Filters" and "Whitelist" tabs (I can see the backup with the name of the other instance), however "3rd-party Filters" and "My Rules" do not show the uploaded settings from the other instance.

I checked the sync-log, and similar to @sander85 , I see errors such as below. Maybe we need to lower maxChunkSize even more?

1506511251900	Sync.Engine.Extension-Storage	ERROR	Syncing uBlock0@raymondhill.net: request failed: Error: HTTP 507; Error: HTTP 507 Insufficient Storage: Resource access is forbidden for this user (Maximum bytes per object exceeded (8379 > 8192 Bytes.) (resource://services-common/kinto-http-client.js:2353:21) JS Stack trace: formatResponse@kinto-http-client.js:2376:21 < processResponse@kinto-http-client.js:2351:14 < waitForSyncCallback@async.js:98:7 < makeSpinningCallback/callback.wait@async.js:168:27 < promiseSpinningly@async.js:234:12 < _sync@extension-storage.js:41:12 < WrappedNotify@util.js:160:21 < sync@engines.js:723:5 < _syncEngine@enginesync.js:219:7 < sync@enginesync.js:166:15 < onNotify@service.js:1081:7 < WrappedNotify@util.js:160:21 < WrappedLock@util.js:116:16 < _lockedSync@service.js:1071:12 < sync/<@service.js:1063:7 < WrappedCatch@util.js:91:16 < sync@service.js:1052:5
1506511251900	Sync.Engine.Extension-Storage	WARN	Syncing failed: Error: HTTP 507; Error: HTTP 507 Insufficient Storage: Resource access is forbidden for this user (Maximum bytes per object exceeded (8379 > 8192 Bytes.) (resource://services-common/kinto-http-client.js:2353:21) JS Stack trace: formatResponse@kinto-http-client.js:2376:21 < processResponse@kinto-http-client.js:2351:14 < waitForSyncCallback@async.js:98:7 < makeSpinningCallback/callback.wait@async.js:168:27 < promiseSpinningly@async.js:234:12 < _sync@extension-storage.js:41:12 < WrappedNotify@util.js:160:21 < sync@engines.js:723:5 < _syncEngine@enginesync.js:219:7 < sync@enginesync.js:166:15 < onNotify@service.js:1081:7 < WrappedNotify@util.js:160:21 < WrappedLock@util.js:116:16 < _lockedSync@service.js:1071:12 < sync/<@service.js:1063:7 < WrappedCatch@util.js:91:16 < sync@service.js:1052:5

[collinbarrett]

Updates after getting to work and trying with uBlock Origin v1.14.10:

• Restoring from another instance via cloud sync works to Firefox Developer v57.03b in all tabs.
• However, it only works restoring to "My Filters" and "Whitelist" tabs in Firefox 55.0.3.

Let me know what other debugging info I can provide to assist with this issue.

[gorhill]

uBO 1.14.10 does not have the fix here, tests should be done with 1.14.11rc?.

I lowered the ratio to 0.5, and I will push this to 1.14.11rc4 today.

[collinbarrett]

I just installed 1.14.11rc5 in both of the following on the same Windows 10 machine:

• Firefox Developer Edition 57.0b3
• Firefox 55.0.3

I re-named the device in uBO on Developer Edition (DE) to something unique, then uploaded all four tabs of settings to the cloud. I forced a sync in the menu of DE. Once that finished, I forced a sync in regular Firefox. As mentioned earlier, I only see the new uploads available to download in uBO in regular Firefox on the "My Filters" and "Whitelist" tabs. The "3rd-party Filters" and "My Rules" tabs are blank in the blue cloud area at the top indicating there was no synced data to download. I believe the lines below are the related errors. Let me know how I can help debug further.

1506533200100	Sync.Engine.Extension-Storage	ERROR	Syncing uBlock0@raymondhill.net: request failed: Error: HTTP 507; Error: HTTP 507 Insufficient Storage: Resource access is forbidden for this user (Maximum bytes per object exceeded (8487 > 8192 Bytes.) (resource://services-common/kinto-http-client.js:2353:21) JS Stack trace: formatResponse@kinto-http-client.js:2376:21 < processResponse@kinto-http-client.js:2351:14 < waitForSyncCallback@async.js:98:7 < makeSpinningCallback/callback.wait@async.js:168:27 < promiseSpinningly@async.js:234:12 < _sync@extension-storage.js:41:12 < WrappedNotify@util.js:160:21 < sync@engines.js:723:5 < _syncEngine@enginesync.js:219:7 < sync@enginesync.js:166:15 < onNotify@service.js:1081:7 < WrappedNotify@util.js:160:21 < WrappedLock@util.js:116:16 < _lockedSync@service.js:1071:12 < sync/<@service.js:1063:7 < WrappedCatch@util.js:91:16 < sync@service.js:1052:5
1506533200100	Sync.Engine.Extension-Storage	WARN	Syncing failed: Error: HTTP 507; Error: HTTP 507 Insufficient Storage: Resource access is forbidden for this user (Maximum bytes per object exceeded (8487 > 8192 Bytes.) (resource://services-common/kinto-http-client.js:2353:21) JS Stack trace: formatResponse@kinto-http-client.js:2376:21 < processResponse@kinto-http-client.js:2351:14 < waitForSyncCallback@async.js:98:7 < makeSpinningCallback/callback.wait@async.js:168:27 < promiseSpinningly@async.js:234:12 < _sync@extension-storage.js:41:12 < WrappedNotify@util.js:160:21 < sync@engines.js:723:5 < _syncEngine@enginesync.js:219:7 < sync@enginesync.js:166:15 < onNotify@service.js:1081:7 < WrappedNotify@util.js:160:21 < WrappedLock@util.js:116:16 < _lockedSync@service.js:1071:12 < sync/<@service.js:1063:7 < WrappedCatch@util.js:91:16 < sync@service.js:1052:5

[gorhill]

That makes little sense, I lowered a lot the ratio to the point where the little difference you reported in #3006 (comment) should be amply covered.

Maybe Firefox is still trying to push the too big data tried from a previous run? What if you push first empty set of data in Firefox 55 in the 3rd-party Filters" and "My Rules", then force a sync. Afterward, try again to push from Dev edition, and see if now you can finally import into FF 55.

[collinbarrett]

Good idea. That solved it. Thanks.

[gorhill]

So if this worked, then I should add code to immediately delete whatever uBO tried to push in the event of failure, so as to not cause the case of perma failure you suffered.

[collinbarrett]

Hmm, sounds reasonable with my limited knowledge of the uBO codebase.

[gorhill]

This won't fix past cases of push-failure however, apparently Firefox will try to push again whatever had fail in the past without uBO being involved.

So below are the reference steps to work around the issue.

For whatever dashboard pane suffering the issue (most likely those which hold lot of custom data):

1. Clear all the data (save it somewhere temporarily, like in a text file).
2. Apply change.
3. Push (the now empty pane) to the cloud.
4. Force a Sync cycle.
5. Optionally, one of the following:
   • Push to the cloud:
     1. Paste the save data back in the pane.
     2. Apply changes.
     3. Push to the cloud.
     4. Force a Sync cycle.
   • Pull from the cloud:
     1. Force reload the pane (i.e. switching to another pane and back)
     2. Pull from the cloud.
     3. Apply changes.

[juhakoho]

I'm an active user of uMatrix and I assume this issue (or a similar one) affects it (version 1.0.1rc2) too but there is no issue tracker where to report it.

The problem is that only the upload button is available but nothing happens when pressed. Tested with FF56 and FF57 beta and with uMatrix versions 1.0.1rc1 and rc2. I also tried the work around steps above with no success.

[gorhill]

@juhakoho fixed in rc3, which I just published on AMO.

[juhakoho]

Thanks @gorhill ! rc3 is working just perfectly!