Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: bypass DOM storage quota #15596

Merged
merged 4 commits into from Nov 12, 2018

Conversation

Projects
None yet
5 participants
@jacobq
Copy link
Contributor

commented Nov 6, 2018

Description of Change

This updates the dom_storage_map patch to include bypassing the 2 additional locations of quota checking. It also adds a test to ensure that at least 50MiB of data can be stored in localStorage and sessionStorage (I'm surprised that I didn't find something like that already in place. Perhaps that's how the regression sneaked in.) Since the usual limit is about 10MiB the test will fail if the patch has not been applied or is not working.
Fixes #8337.

cc @MarshallOfSound you mentioned you might be willing to help land this. (Thanks in advance!)
cc @zcbenz It looks like you wrote the original patch. (Thank you! That helped me understand how electron's build system applies chromium patches)
cc @asinbow You pointed out the additional checks that needed to be disabled. (Thank you!)

Checklist

  • PR description included and stakeholders cc'd
  • npm test passes (CI is green)
  • tests are changed or added
  • relevant documentation is changed or added (see note below)
  • PR title follows semantic commit guidelines (once WIP prefix is removed)

Note about documentation:
Since this is fixing a regression (doing what the previous patch used to do), I don't think there's any need to change documentation. However, it might make sense to record this behavior in the documentation somewhere (I couldn't find it). Perhaps in the FAQ where localStorage is mentioned.

Note about running tests locally:
It seems that I can't run the test suite locally, though I can build a release and manually test in the console. Here's what I get on my (Debian 9.5) machine when I try to build the debug version (needed for running tests, right?) It seems like my system's libc version does not match the required one.

$ ninja -C out/Debug electron
ninja: Entering directory `out/Debug'
[7817/18095] ACTION //components/url_formatter/top_domains:generate_top_domains_trie(//build/toolchain/linux:clang_x64)
FAILED: gen/components/url_formatter/top_domains/alexa_domains-trie-inc.cc 
python ../../build/gn_run_binary.py top_domain_generator /home/jquant/tmp/electron-gn/src/components/url_formatter/top_domains/alexa_domains.skeletons /home/jquant/tmp/electron-gn/src/components/url_formatter/top_domains/top_domains_trie.template gen/components/url_formatter/top_domains/alexa_domains-trie-inc.cc
./top_domain_generator: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.27' not found (required by /home/jquant/tmp/electron-gn/src/out/Debug/./libbase.so)
top_domain_generator failed with exit code 1
[7818/18095] ACTION //net/http:generate_transport_security_state(//build/toolchain/linux:clang_x64)
FAILED: gen/net/http/transport_security_state_static.h 
python ../../build/gn_run_binary.py transport_security_state_generator ../../net/http/transport_security_state_static.json ../../net/http/transport_security_state_static.pins ../../net/http/transport_security_state_static.template gen/net/http/transport_security_state_static.h
./transport_security_state_generator: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.27' not found (required by /home/jquant/tmp/electron-gn/src/out/Debug/./libbase.so)
transport_security_state_generator failed with exit code 1
[7826/18095] CXX obj/third_party/ots/ots/ots.o
ninja: build stopped: subcommand failed.

Release Notes

notes: Fixes regression regarding localStorage quota not being bypassed.

@jacobq jacobq requested a review from as a code owner Nov 6, 2018

@welcome

This comment has been minimized.

Copy link

commented Nov 6, 2018

💖 Thanks for opening this pull request! 💖

We use semantic commit messages to streamline the release process. Before your pull request can be merged, you should update your pull request title to start with a semantic prefix.

Examples of commit messages with semantic prefixes:

  • fix: don't overwrite prevent_default if default wasn't prevented
  • feat: add app.isPackaged() method
  • docs: app.isDefaultProtocolClient is now available on Linux

Things that will help get your PR across the finish line:

  • Follow the JavaScript, C++, and Python coding style.
  • Run npm run lint locally to catch formatting errors earlier.
  • Document any user-facing changes you've made following the documentation styleguide.
  • Include tests when adding/changing behavior.
  • Include screenshots and animated GIFs whenever possible.

We get a lot of pull requests on this repo, so please be patient and we will get back to you as soon as we can.

@deepak1556

This comment has been minimized.

Copy link
Member

commented Nov 6, 2018

Although the patch looks good, I would like to get a better understanding of the side effects from changing the rate limiter for writes, storage-dev would be the best place to get more thoughts on this approach.

@jacobq

This comment has been minimized.

Copy link
Contributor Author

commented Nov 6, 2018

I would like to get a better understanding of the side effects from changing the rate limiter for writes, storage-dev

I just threw that in there "just in case" as, once again, it seems like something that shouldn't be restricted in Electron. If it will help land this sooner I will take that out.

@jacobq jacobq force-pushed the jacobq:master branch 2 times, most recently from 1912cb2 to a9938de Nov 6, 2018

@jacobq jacobq changed the title WIP: fix: Bypass localStorage quota (ref #8337) fix: bypass DOM storage quota (fix #8337) Nov 6, 2018

@jacobq

This comment has been minimized.

Copy link
Contributor Author

commented Nov 6, 2018

@deepak1556 I removed the change about which you had concerns. Would you mind reviewing again?

@MarshallOfSound

This comment has been minimized.

Copy link
Member

commented Nov 7, 2018

needed for running tests, right?

@jacobq Actually on some platforms you can't run the tests on the debug build 😄, we have a testing build config that we use to run the tests

@deepak1556
Copy link
Member

left a comment

LGTM, can you add a description in the patch file for why it is needed, #15612 is a good example. Thanks!

@jacobq jacobq force-pushed the jacobq:master branch from a9938de to 9b57e38 Nov 7, 2018

@jacobq

This comment has been minimized.

Copy link
Contributor Author

commented Nov 7, 2018

@deepak1556 Sure, description starts here. More discussion may also be found in the related issues.

@jacobq

This comment has been minimized.

Copy link
Contributor Author

commented Nov 7, 2018

Could this also be backported to 3-0-x?

@jacobq

This comment has been minimized.

Copy link
Contributor Author

commented Nov 7, 2018

Note: failed win-ia32 test appears unrelated. Would one of the releasers please restart the failed build(s)?

@jacobq

This comment has been minimized.

Copy link
Contributor Author

commented Nov 7, 2018

Similar to the win-ia32 false positive, would one of the reviewers please restart the build for mas-testing-tests and osx-testing-tests. I don't think those failures were caused by this PR.

@deepak1556
Copy link
Member

left a comment

LGTM , Thanks!

@deepak1556 deepak1556 requested review from nornagon and alexeykuzmin Nov 8, 2018

@alexeykuzmin alexeykuzmin changed the title fix: bypass DOM storage quota (fix #8337) fix: bypass DOM storage quota Nov 10, 2018

@alexeykuzmin
Copy link
Contributor

left a comment

👍

Show resolved Hide resolved spec/chromium-spec.js Outdated
Show resolved Hide resolved spec/chromium-spec.js Outdated
@@ -937,6 +937,23 @@ describe('chromium feature', () => {
})

describe('storage', () => {
describe('DOM storage quota override', () => {
['localStorage', 'sessionStorage'].forEach((storageName) => {
it(`can store at least 50MiB in ${storageName}`, (done) => {

This comment has been minimized.

Copy link
@alexeykuzmin

alexeykuzmin Nov 10, 2018

Contributor

A done callback is not needed for synchronous tests, please remove it.

@alexeykuzmin

This comment has been minimized.

Copy link
Contributor

commented Nov 10, 2018

Could this also be backported to 3-0-x?

@jacobq
Yes, but not automatically. We have significantly changed a layout of the Electron repositories between versions 3 and 4.
If you want to see this change in the Electron 3.x release line, please open a PR against the electron-3-0-x branch of the https://github.com/electron/libchromiumcontent repo, changing the same patch file. Thank you.

refactor: change assert to expect in DOM storage test
Co-Authored-By: jacobq <jacobq@gmail.com>

@jacobq jacobq force-pushed the jacobq:master branch from 1f75a7d to f9193d5 Nov 11, 2018

@jacobq

This comment has been minimized.

Copy link
Contributor Author

commented Nov 11, 2018

@alexeykuzmin OK, I just added some more commits and think I got all the changes you requested. Once this lands I'll open a separate PR for 3.0.x backport.

Edit: Looks like it won't be quite as simple as committing the same patch file because the hash of the files being patched has changed. So I think I'll have to regenerate it against chrome 66, not a biggy, but not a 5 minute ordeal for me either.

@jacobq

This comment has been minimized.

Copy link
Contributor Author

commented Nov 12, 2018

Hello again...would someone please re-run the 2 failed CI jobs? (mas-testing-tests & osx-testing-tests)

@alexeykuzmin
Copy link
Contributor

left a comment

@jacobq Thank you for the fix.
I just restarted those CI jobs.

@alexeykuzmin alexeykuzmin merged commit b21dbdb into electron:master Nov 12, 2018

24 checks passed

Absolute Zero
Artifact Comparison No Changes
Details
Backportable? - 4-0-x Clean Backport
Details
Semantic Pull Request ready to be squashed
Details
WIP ready for review
Details
appveyor: win-ia32-testing-pr AppVeyor build succeeded
Details
appveyor: win-x64-testing-pr AppVeyor build succeeded
Details
ci/circleci: linux-arm-debug Your tests passed on CircleCI!
Details
ci/circleci: linux-arm-testing Your tests passed on CircleCI!
Details
ci/circleci: linux-arm64-debug Your tests passed on CircleCI!
Details
ci/circleci: linux-arm64-testing Your tests passed on CircleCI!
Details
ci/circleci: linux-checkout Your tests passed on CircleCI!
Details
ci/circleci: linux-ia32-debug Your tests passed on CircleCI!
Details
ci/circleci: linux-ia32-testing Your tests passed on CircleCI!
Details
ci/circleci: linux-ia32-testing-tests Your tests passed on CircleCI!
Details
ci/circleci: linux-x64-debug Your tests passed on CircleCI!
Details
ci/circleci: linux-x64-testing Your tests passed on CircleCI!
Details
ci/circleci: linux-x64-testing-tests Your tests passed on CircleCI!
Details
ci/circleci: mas-testing Your tests passed on CircleCI!
Details
ci/circleci: mas-testing-tests Your tests passed on CircleCI!
Details
ci/circleci: osx-testing Your tests passed on CircleCI!
Details
ci/circleci: osx-testing-tests Your tests passed on CircleCI!
Details
electron-lint Build #20181111.6 succeeded
Details
release-notes Release notes found
@release-clerk

This comment has been minimized.

Copy link

commented Nov 12, 2018

Release Notes Persisted

Fixes regression regarding localStorage quota not being bypassed.

@welcome

This comment has been minimized.

Copy link

commented Nov 12, 2018

Congrats on merging your first pull request! 🎉🎉🎉

@trop

This comment has been minimized.

Copy link
Contributor

commented Nov 12, 2018

I have automatically backported this PR to "4-0-x", please check out #15686

// 25 * 2^20 UTF-16 characters will require 50MiB
const arraySize = 25 * Math.pow(2, 20)
storage[testKeyName] = new Array(arraySize).fill('X').join('')
expect(storage[testKeyName]).to.have.lengthOf(arraySize)

This comment has been minimized.

Copy link
@jacobq

jacobq May 22, 2019

Author Contributor

@deepak1556 I think this test is actually not working correctly. It appears to pass because no exceptions are thrown, but the data isn't really being persisted, and this can be seen by separating the read and write operations by a setTimeout, page refresh, etc. See #13465 (comment) . Any idea how we can fix this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.