Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sync is bogus with the new version of uBlock #3006

Closed
sander85 opened this Issue Sep 11, 2017 · 25 comments

Comments

Projects
None yet
4 participants
@sander85
Copy link
Contributor

sander85 commented Sep 11, 2017

Describe the issue

After migrating to web extension of the uBlock I can't sync my devices anymore. It seems to be able to sync (download) only when I have uploaded something but this would erase all the changes that I have uploaded from the other device.

One or more specific URLs where the issue occurs

moz-extension://cad1c7f1-1273-499c-a9a6-625f965628d9/dashboard.html#dyna-rules.html

Screenshot in which the issue can be seen

only_upload_button

Steps for anyone to reproduce the issue

  1. Migrate to new extension.
  2. Try to pull some new changes from Sync.
  3. Only upload button is shown.

Your settings

Sync is enabled, not sure if it's enabled by default.

  • OS/version: Linux (Mageia 6)
  • Browser/version: Firefox 55.0.3 (64-bit)
  • uBlock Origin version: v1.14.4
Your filter lists

I don't think these are relevant in this bug.

Your custom filters (if any)

Same as above.

@gorhill

This comment has been minimized.

Copy link
Owner

gorhill commented Sep 11, 2017

I don't understand your issue. In your steps-to-reproduce, there is no step which says "Click 'Export to cloud storage'", hence of course there won't be anything to import from cloud storage.

If your issue is that your cloud storage was filled with legacy uBO data, then it's expected you can't import it with webext version, the Sync data is stored in a different way with webext version. If you are unhappy with this, you will have to close the issue here and open one at bugzilla.

@sander85

This comment has been minimized.

Copy link
Contributor Author

sander85 commented Sep 11, 2017

The problem is that I exported it to cloud from another device but am unable to import on other devices. This worked with the legacy version of uBlock.

@gorhill

This comment has been minimized.

Copy link
Owner

gorhill commented Sep 11, 2017

I exported it to cloud from another device

You exported with what version of uBO? What Firefox version was it?

I did try the Sync feature, and it worked fine here.

@sander85

This comment has been minimized.

Copy link
Contributor Author

sander85 commented Sep 11, 2017

uBO version and Firefox are the same on both systems. But I found this from about:sync-log:

1505156092353	Sync.Engine.Extension-Storage	ERROR	Syncing uBlock0@raymondhill.net: request failed: Error: HTTP 507; Error: HTTP 507 Insufficient Storage: Resource access is forbidden for this user (Maximum bytes per object exceeded (8763 > 8192 Bytes.) (resource://services-common/kinto-http-client.js:2353:21) JS Stack trace: formatResponse@kinto-http-client.js:2376:21 < processResponse@kinto-http-client.js:2351:14 < waitForSyncCallback@async.js:98:7 < makeSpinningCallback/callback.wait@async.js:168:27 < promiseSpinningly@async.js:234:12 < _sync@extension-storage.js:41:12 < WrappedNotify@util.js:160:21 < sync@engines.js:723:5 < _syncEngine@enginesync.js:219:7 < sync@enginesync.js:166:15 < onNotify@service.js:1081:7 < WrappedNotify@util.js:160:21 < WrappedLock@util.js:116:16 < _lockedSync@service.js:1071:12 < sync/<@service.js:1063:7 < WrappedCatch@util.js:91:16 < sync@service.js:1052:5
1505156092354	Sync.Engine.Extension-Storage	WARN	Syncing failed: Error: HTTP 507; Error: HTTP 507 Insufficient Storage: Resource access is forbidden for this user (Maximum bytes per object exceeded (8763 > 8192 Bytes.) (resource://services-common/kinto-http-client.js:2353:21) JS Stack trace: formatResponse@kinto-http-client.js:2376:21 < processResponse@kinto-http-client.js:2351:14 < waitForSyncCallback@async.js:98:7 < makeSpinningCallback/callback.wait@async.js:168:27 < promiseSpinningly@async.js:234:12 < _sync@extension-storage.js:41:12 < WrappedNotify@util.js:160:21 < sync@engines.js:723:5 < _syncEngine@enginesync.js:219:7 < sync@enginesync.js:166:15 < onNotify@service.js:1081:7 < WrappedNotify@util.js:160:21 < WrappedLock@util.js:116:16 < _lockedSync@service.js:1071:12 < sync/<@service.js:1063:7 < WrappedCatch@util.js:91:16 < sync@service.js:1052:5

Not sure if it's Firefox/Sync to blame or if uBO should send smaller objects?

@gorhill

This comment has been minimized.

Copy link
Owner

gorhill commented Sep 11, 2017

I internally work with chunks of max size 6,144 character-long string segments, which to me seems safe enough taking into account that this is put through JSON.stringify. What sort of string derived from a ruleset would cause a 6,144 character-long string to expand to 8,763 bytes once JSON.stringify'ied?

@sander85

This comment has been minimized.

Copy link
Contributor Author

sander85 commented Sep 11, 2017

This is the list that I have: https://sander85.eu/paste/view/9d08bf45

I don't have any other custom rules.

@gorhill

This comment has been minimized.

Copy link
Owner

gorhill commented Sep 11, 2017

Ok, so FF encrypts and then encodes the result as base64. Unfortunately, from what I see the per-item data size limit applies after the encrypting/encoding, so this makes it more difficult to find a safe max chunk size -- both the encryption and encoding add their own data size overhead, and add to this FF's own information (like last_time_modified) is added to the original data. It could be argued with Firefox people that the limit should apply before the encrypting/encoding.

@gorhill gorhill closed this in 0d18d99 Sep 11, 2017

@gorhill

This comment has been minimized.

Copy link
Owner

gorhill commented Sep 11, 2017

I just lowered the max chunk size for now to at least avoid failure, I verified it works with your ruleset.

@gorhill

This comment has been minimized.

Copy link
Owner

gorhill commented Sep 11, 2017

If you are willing to use the dev build, fix is in 1.14.9b1 in AMO's dev channel.

@sander85

This comment has been minimized.

Copy link
Contributor Author

sander85 commented Sep 12, 2017

Thanks a lot! Will test ASAP :)

@sander85

This comment has been minimized.

Copy link
Contributor Author

sander85 commented Sep 13, 2017

Weird, I'm still seeing this error:
1505327718411 Sync.ErrorHandler DEBUG extension-storage failed: Error: HTTP 507; Error: HTTP 507 Insufficient Storage: Resource access is forbidden for this user (Maximum bytes per object exceeded (8763 > 8192 Bytes.)

This is with 1.14.9b1

@gorhill

This comment has been minimized.

Copy link
Owner

gorhill commented Sep 13, 2017

I just tested again and it worked on my side:

a

On the left is Firefox 55 with a new profile + uBO dev build + all your rules. On the right is Firefox Nightly, in which you can see I was able to see the rules from the cloud storage, which I successfully imported. Here are the steps I went through:

  1. Log in to Sync in both FF 55 and FF Nightly.
  2. In FF 55, renamed the device "New Temp Profile" in uBO
  3. In FF 55, sent the rules to the cloud
  4. In FF 55, clicked the sync button to force a sync
  5. In FF Nightly, clicked the sync button to force a sync
  6. In FF Nightly, I forced a refresh of the "My rules" pane to force uBO to read whatever data is available in cloud storage
  7. Result: FF Nightly showed the data that was sent from FF 55
  8. I could then import all the rules, as seen in the screenshot
@sander85

This comment has been minimized.

Copy link
Contributor Author

sander85 commented Sep 14, 2017

It seems I had to push the button to upload the rules again and after that it now works. Superb job!

@collinbarrett

This comment has been minimized.

Copy link

collinbarrett commented Sep 27, 2017

Hmm, I am seeing the same issue right now. I am using Firefox Developer v57.03b and Firefox 55.0.3. I installed the latest dev build 1.14.11rc3 of uBlock Origin.

Sync works for "My Filters" and "Whitelist" tabs (I can see the backup with the name of the other instance), however "3rd-party Filters" and "My Rules" do not show the uploaded settings from the other instance.

I checked the sync-log, and similar to @sander85 , I see errors such as below. Maybe we need to lower maxChunkSize even more?

1506511251900	Sync.Engine.Extension-Storage	ERROR	Syncing uBlock0@raymondhill.net: request failed: Error: HTTP 507; Error: HTTP 507 Insufficient Storage: Resource access is forbidden for this user (Maximum bytes per object exceeded (8379 > 8192 Bytes.) (resource://services-common/kinto-http-client.js:2353:21) JS Stack trace: formatResponse@kinto-http-client.js:2376:21 < processResponse@kinto-http-client.js:2351:14 < waitForSyncCallback@async.js:98:7 < makeSpinningCallback/callback.wait@async.js:168:27 < promiseSpinningly@async.js:234:12 < _sync@extension-storage.js:41:12 < WrappedNotify@util.js:160:21 < sync@engines.js:723:5 < _syncEngine@enginesync.js:219:7 < sync@enginesync.js:166:15 < onNotify@service.js:1081:7 < WrappedNotify@util.js:160:21 < WrappedLock@util.js:116:16 < _lockedSync@service.js:1071:12 < sync/<@service.js:1063:7 < WrappedCatch@util.js:91:16 < sync@service.js:1052:5
1506511251900	Sync.Engine.Extension-Storage	WARN	Syncing failed: Error: HTTP 507; Error: HTTP 507 Insufficient Storage: Resource access is forbidden for this user (Maximum bytes per object exceeded (8379 > 8192 Bytes.) (resource://services-common/kinto-http-client.js:2353:21) JS Stack trace: formatResponse@kinto-http-client.js:2376:21 < processResponse@kinto-http-client.js:2351:14 < waitForSyncCallback@async.js:98:7 < makeSpinningCallback/callback.wait@async.js:168:27 < promiseSpinningly@async.js:234:12 < _sync@extension-storage.js:41:12 < WrappedNotify@util.js:160:21 < sync@engines.js:723:5 < _syncEngine@enginesync.js:219:7 < sync@enginesync.js:166:15 < onNotify@service.js:1081:7 < WrappedNotify@util.js:160:21 < WrappedLock@util.js:116:16 < _lockedSync@service.js:1071:12 < sync/<@service.js:1063:7 < WrappedCatch@util.js:91:16 < sync@service.js:1052:5

@gorhill gorhill reopened this Sep 27, 2017

@collinbarrett

This comment has been minimized.

Copy link

collinbarrett commented Sep 27, 2017

Updates after getting to work and trying with uBlock Origin v1.14.10:

  • Restoring from another instance via cloud sync works to Firefox Developer v57.03b in all tabs.
  • However, it only works restoring to "My Filters" and "Whitelist" tabs in Firefox 55.0.3.

Let me know what other debugging info I can provide to assist with this issue.

@gorhill

This comment has been minimized.

Copy link
Owner

gorhill commented Sep 27, 2017

uBO 1.14.10 does not have the fix here, tests should be done with 1.14.11rc?.

I lowered the ratio to 0.5, and I will push this to 1.14.11rc4 today.

@gorhill gorhill closed this in f10fb29 Sep 27, 2017

@collinbarrett

This comment has been minimized.

Copy link

collinbarrett commented Sep 27, 2017

I just installed 1.14.11rc5 in both of the following on the same Windows 10 machine:

  • Firefox Developer Edition 57.0b3
  • Firefox 55.0.3

I re-named the device in uBO on Developer Edition (DE) to something unique, then uploaded all four tabs of settings to the cloud. I forced a sync in the menu of DE. Once that finished, I forced a sync in regular Firefox. As mentioned earlier, I only see the new uploads available to download in uBO in regular Firefox on the "My Filters" and "Whitelist" tabs. The "3rd-party Filters" and "My Rules" tabs are blank in the blue cloud area at the top indicating there was no synced data to download. I believe the lines below are the related errors. Let me know how I can help debug further.

1506533200100	Sync.Engine.Extension-Storage	ERROR	Syncing uBlock0@raymondhill.net: request failed: Error: HTTP 507; Error: HTTP 507 Insufficient Storage: Resource access is forbidden for this user (Maximum bytes per object exceeded (8487 > 8192 Bytes.) (resource://services-common/kinto-http-client.js:2353:21) JS Stack trace: formatResponse@kinto-http-client.js:2376:21 < processResponse@kinto-http-client.js:2351:14 < waitForSyncCallback@async.js:98:7 < makeSpinningCallback/callback.wait@async.js:168:27 < promiseSpinningly@async.js:234:12 < _sync@extension-storage.js:41:12 < WrappedNotify@util.js:160:21 < sync@engines.js:723:5 < _syncEngine@enginesync.js:219:7 < sync@enginesync.js:166:15 < onNotify@service.js:1081:7 < WrappedNotify@util.js:160:21 < WrappedLock@util.js:116:16 < _lockedSync@service.js:1071:12 < sync/<@service.js:1063:7 < WrappedCatch@util.js:91:16 < sync@service.js:1052:5
1506533200100	Sync.Engine.Extension-Storage	WARN	Syncing failed: Error: HTTP 507; Error: HTTP 507 Insufficient Storage: Resource access is forbidden for this user (Maximum bytes per object exceeded (8487 > 8192 Bytes.) (resource://services-common/kinto-http-client.js:2353:21) JS Stack trace: formatResponse@kinto-http-client.js:2376:21 < processResponse@kinto-http-client.js:2351:14 < waitForSyncCallback@async.js:98:7 < makeSpinningCallback/callback.wait@async.js:168:27 < promiseSpinningly@async.js:234:12 < _sync@extension-storage.js:41:12 < WrappedNotify@util.js:160:21 < sync@engines.js:723:5 < _syncEngine@enginesync.js:219:7 < sync@enginesync.js:166:15 < onNotify@service.js:1081:7 < WrappedNotify@util.js:160:21 < WrappedLock@util.js:116:16 < _lockedSync@service.js:1071:12 < sync/<@service.js:1063:7 < WrappedCatch@util.js:91:16 < sync@service.js:1052:5
@gorhill

This comment has been minimized.

Copy link
Owner

gorhill commented Sep 27, 2017

That makes little sense, I lowered a lot the ratio to the point where the little difference you reported in #3006 (comment) should be amply covered.

Maybe Firefox is still trying to push the too big data tried from a previous run? What if you push first empty set of data in Firefox 55 in the 3rd-party Filters" and "My Rules", then force a sync. Afterward, try again to push from Dev edition, and see if now you can finally import into FF 55.

@collinbarrett

This comment has been minimized.

Copy link

collinbarrett commented Sep 27, 2017

Good idea. That solved it. Thanks.

@gorhill

This comment has been minimized.

Copy link
Owner

gorhill commented Sep 27, 2017

So if this worked, then I should add code to immediately delete whatever uBO tried to push in the event of failure, so as to not cause the case of perma failure you suffered.

@gorhill gorhill reopened this Sep 27, 2017

@collinbarrett

This comment has been minimized.

Copy link

collinbarrett commented Sep 27, 2017

Hmm, sounds reasonable with my limited knowledge of the uBO codebase.

@gorhill gorhill closed this in a4e61b5 Sep 27, 2017

@gorhill

This comment has been minimized.

Copy link
Owner

gorhill commented Sep 27, 2017

This won't fix past cases of push-failure however, apparently Firefox will try to push again whatever had fail in the past without uBO being involved.

So below are the reference steps to work around the issue.

For whatever dashboard pane suffering the issue (most likely those which hold lot of custom data):

  1. Clear all the data (save it somewhere temporarily, like in a text file).
  2. Apply change.
  3. Push (the now empty pane) to the cloud.
  4. Force a Sync cycle.
  5. Optionally, one of the following:
    • Push to the cloud:
      1. Paste the save data back in the pane.
      2. Apply changes.
      3. Push to the cloud.
      4. Force a Sync cycle.
    • Pull from the cloud:
      1. Force reload the pane (i.e. switching to another pane and back)
      2. Pull from the cloud.
      3. Apply changes.
@juhakoho

This comment has been minimized.

Copy link

juhakoho commented Sep 28, 2017

I'm an active user of uMatrix and I assume this issue (or a similar one) affects it (version 1.0.1rc2) too but there is no issue tracker where to report it.

The problem is that only the upload button is available but nothing happens when pressed. Tested with FF56 and FF57 beta and with uMatrix versions 1.0.1rc1 and rc2. I also tried the work around steps above with no success.

gorhill added a commit to gorhill/uMatrix that referenced this issue Sep 28, 2017

@gorhill

This comment has been minimized.

Copy link
Owner

gorhill commented Sep 28, 2017

@juhakoho fixed in rc3, which I just published on AMO.

@juhakoho

This comment has been minimized.

Copy link

juhakoho commented Sep 28, 2017

Thanks @gorhill ! rc3 is working just perfectly!

gorhill added a commit that referenced this issue Sep 29, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.