Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Optimizer] Fix a stack overflow with watch_cache when it attempts to delete very large folders. #54457

Merged
merged 5 commits into from Jan 11, 2020

Conversation

@chrisdavies
Copy link
Contributor

chrisdavies commented Jan 10, 2020

This happens occasionally when the .cache/ui_bundles has a lot of files. The reason is that globby spreads all of the file paths as arguments to a function call: sindresorhus/globby#141

This is a quick workaround for the offending call site.

You can reproduce the bug by running node and then running a script like this (after replacing the placeholders):

const fs = require('fs');

for (let i = 1; i < 80000; ++i) {
  fs.writeFileSync(
    `PREFIX/kibana/data/built_assets/.cache/ui_bundles/BUNDLE_PATH/babel/test_${i}.json`,
    '{}'
  );
}
  • Replace PREFIX/ with whatever is appropriate for your system
  • Replace BUNDLE_PATH with whatever the existing path is on your system
  • Change the yarn.lock file in some way yarn add ramda or something
  • Run node scripts/functional_tests_server to see the error.
chrisdavies added 2 commits Jan 10, 2020
attempts to delete very large folders.
@chrisdavies chrisdavies requested a review from elastic/kibana-operations as a code owner Jan 10, 2020
@chrisdavies chrisdavies requested a review from spalger Jan 10, 2020
@chrisdavies chrisdavies deleted the chrisdavies:optimizer/max_stack_size_bug branch Jan 10, 2020
@chrisdavies chrisdavies reopened this Jan 10, 2020
@chrisdavies

This comment has been minimized.

Copy link
Contributor Author

chrisdavies commented Jan 10, 2020

retest

@chrisdavies

This comment has been minimized.

Copy link
Contributor Author

chrisdavies commented Jan 10, 2020

@elasticmachine merge upstream

@chrisdavies chrisdavies added the v7.6.0 label Jan 10, 2020
@kibanamachine

This comment has been minimized.

Copy link

kibanamachine commented Jan 10, 2020

💚 Build Succeeded

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

Copy link
Member

tylersmalley left a comment

LGTM - thanks the fix!

@chrisdavies chrisdavies merged commit 8ef5609 into elastic:master Jan 11, 2020
56 checks passed
56 checks passed
Assign a PR to project based on label
Details
API integration tests node scripts/functional_tests --config test/api_integration/config.js --bail --debug
Details
Browser tests yarn run grunt test:browser-ci
Details
Build kbn_tp_sample_panel_action yarn build
Details
CLA All commits in pull request signed
Details
Check core API changes node scripts/check_core_api_changes
Details
Check file casing node scripts/check_file_casing --quiet
Details
Check licenses node scripts/check_licenses --dev
Details
Example functional tests node scripts/functional_tests --config test/examples/config.js --bail --debug
Details
Firefox smoke test node scripts/functional_tests --bail --debug --kibana-install-dir /dev/shm/workspace/kibana/build/oss/kibana-8.0.0-SNAPSHOT-linux-x86_64 --include-tag smoke --config test/functional/config.firefox.js
Details
Functional tests / Group 1 yarn run grunt run:functionalTests_ciGroup1
Details
Functional tests / Group 10 yarn run grunt run:functionalTests_ciGroup10
Details
Functional tests / Group 11 yarn run grunt run:functionalTests_ciGroup11
Details
Functional tests / Group 12 yarn run grunt run:functionalTests_ciGroup12
Details
Functional tests / Group 2 yarn run grunt run:functionalTests_ciGroup2
Details
Functional tests / Group 3 yarn run grunt run:functionalTests_ciGroup3
Details
Functional tests / Group 4 yarn run grunt run:functionalTests_ciGroup4
Details
Functional tests / Group 5 yarn run grunt run:functionalTests_ciGroup5
Details
Functional tests / Group 6 yarn run grunt run:functionalTests_ciGroup6
Details
Functional tests / Group 7 yarn run grunt run:functionalTests_ciGroup7
Details
Functional tests / Group 8 yarn run grunt run:functionalTests_ciGroup8
Details
Functional tests / Group 9 yarn run grunt run:functionalTests_ciGroup9
Details
Internationalization check node scripts/i18n_check --ignore-missing
Details
Interpreter functional tests node scripts/functional_tests --config test/interpreter_functional/config.ts --bail --debug --kibana-install-dir /dev/shm/workspace/kibana/build/oss/kibana-8.0.0-SNAPSHOT-linux-x86_64-2
Details
Jest integration tests yarn run grunt test:jest_integration
Details
Jest tests yarn run grunt test:jest
Details
Kibana accessibility tests node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/kibana/build/oss/kibana-8.0.0-SNAPSHOT-linux-x86_64 --config test/accessibility/config.ts
Details
Mocha tests node scripts/mocha
Details
Plugin functional tests node scripts/functional_tests --config test/plugin_functional/config.js --bail --debug
Details
Project tests yarn run grunt test:projects
Details
Type check node scripts/type_check
Details
TypeScript - all files belong to a TypeScript project node scripts/check_ts_projects
Details
Verify NOTICE.txt node scripts/notice --validate
Details
Verify dependency versions yarn run grunt verifyDependencyVersions
Details
X-Pack Chrome Functional tests / Group 1 node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/install/kibana-2 --include-tag ciGroup1
Details
X-Pack Chrome Functional tests / Group 10 node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/install/kibana-11 --include-tag ciGroup10
Details
X-Pack Chrome Functional tests / Group 2 node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/install/kibana-3 --include-tag ciGroup2
Details
X-Pack Chrome Functional tests / Group 3 node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/install/kibana-4 --include-tag ciGroup3
Details
X-Pack Chrome Functional tests / Group 4 node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/install/kibana-5 --include-tag ciGroup4
Details
X-Pack Chrome Functional tests / Group 5 node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/install/kibana-6 --include-tag ciGroup5
Details
X-Pack Chrome Functional tests / Group 6 node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/install/kibana-7 --include-tag ciGroup6
Details
X-Pack Chrome Functional tests / Group 7 node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/install/kibana-8 --include-tag ciGroup7
Details
X-Pack Chrome Functional tests / Group 8 node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/install/kibana-9 --include-tag ciGroup8
Details
X-Pack Chrome Functional tests / Group 9 node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/install/kibana-10 --include-tag ciGroup9
Details
X-Pack Jest node scripts/jest --ci --verbose
Details
X-Pack Karma Tests yarn test:browser
Details
X-Pack SIEM cyclic dependency test node legacy/plugins/siem/scripts/check_circular_deps
Details
X-Pack accessibility tests node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/install/kibana --config test/accessibility/config.ts
Details
X-Pack firefox smoke test node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/install/kibana-1 --include-tag smoke --config test/functional/config.firefox.js
Details
elasticsearch-ci/docs Build finished.
Details
eslint node scripts/eslint --no-cache
Details
kibana-ci Build finished.
Details
prbot:outdated
prbot:release note labels
prbot:release version labels
sasslint node scripts/sasslint
Details
thomasneirynck added a commit to thomasneirynck/kibana that referenced this pull request Jan 12, 2020
gmmorris added a commit to gmmorris/kibana that referenced this pull request Jan 13, 2020
* master: (69 commits)
  [Graph] Fix various a11y issues (elastic#54097)
  Add ApplicationService app status management (elastic#50223)
  logs in one time (elastic#54447)
  Deprecate using `elasticsearch.ssl.certificate` without `elasticsearch.ssl.key` and vice versa (elastic#54392)
  [Optimizer] Fix a stack overflow with watch_cache when it attempts to delete very large folders. (elastic#54457)
  Security - Role Mappings UI (elastic#53620)
  [SIEM] [Detection engine] Permission II (elastic#54292)
  Allow User to Cleanup Repository from UI  (elastic#53047)
  [Detection engine] Some UX for rule creation (elastic#54471)
  share specific instances of some ui packages (elastic#54079)
  [ML] APM modules configs for RUM Javascript and NodeJS (elastic#53792)
  [APM] Delay rendering invalid license notification (elastic#53924)
  [Graph] Improve error message on graph requests (elastic#54230)
  [ILM] Kibana should allow a min_age setting of 0ms in ILM policy phases (elastic#53719)
  Unit Tests for common/lib (elastic#53736)
  [Graph] Only show explorable fields (elastic#54101)
  remove linting rule exception for markdown (elastic#54232)
  [Monitoring] Fetch shard data more efficiently (elastic#54028)
  [Maps] Add hiddenLayers option to embeddable map input (elastic#54355)
  Pass termOrder and hasTermsAgg properties to serializeThresholdWatch function (elastic#54391)
  ...
gmmorris added a commit to gmmorris/kibana that referenced this pull request Jan 13, 2020
* master: (69 commits)
  [Graph] Fix various a11y issues (elastic#54097)
  Add ApplicationService app status management (elastic#50223)
  logs in one time (elastic#54447)
  Deprecate using `elasticsearch.ssl.certificate` without `elasticsearch.ssl.key` and vice versa (elastic#54392)
  [Optimizer] Fix a stack overflow with watch_cache when it attempts to delete very large folders. (elastic#54457)
  Security - Role Mappings UI (elastic#53620)
  [SIEM] [Detection engine] Permission II (elastic#54292)
  Allow User to Cleanup Repository from UI  (elastic#53047)
  [Detection engine] Some UX for rule creation (elastic#54471)
  share specific instances of some ui packages (elastic#54079)
  [ML] APM modules configs for RUM Javascript and NodeJS (elastic#53792)
  [APM] Delay rendering invalid license notification (elastic#53924)
  [Graph] Improve error message on graph requests (elastic#54230)
  [ILM] Kibana should allow a min_age setting of 0ms in ILM policy phases (elastic#53719)
  Unit Tests for common/lib (elastic#53736)
  [Graph] Only show explorable fields (elastic#54101)
  remove linting rule exception for markdown (elastic#54232)
  [Monitoring] Fetch shard data more efficiently (elastic#54028)
  [Maps] Add hiddenLayers option to embeddable map input (elastic#54355)
  Pass termOrder and hasTermsAgg properties to serializeThresholdWatch function (elastic#54391)
  ...
chrisronline added a commit to chrisronline/kibana that referenced this pull request Jan 13, 2020
@tylersmalley tylersmalley added the v8.0.0 label Jan 14, 2020
mistic added a commit that referenced this pull request Jan 14, 2020
… delete very large folders. (#54457) (#54535)

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
@mistic

This comment has been minimized.

Copy link
Member

mistic commented Jan 14, 2020

7.x: 63d0b60

jkelastic added a commit to jkelastic/kibana that referenced this pull request Jan 17, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.