Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: Releasing SQL Lab Celery results #10279

Merged

Conversation

john-bodley
Copy link
Member

@john-bodley john-bodley commented Jul 10, 2020

SUMMARY

Per the Celery documentation in relation to Celery results,

Warning:
Backends use resources to store and transmit results. To ensure that resources are released, you must eventually call get() or forget() on EVERY AsyncResult instance returned after calling a task.

which we aren't doing. Given we actually don't store the SQL Lab payloads in the Celery results backend, holding onto these task result metadata for 24 hours isn't a major concern, however this can result in having thousands of redundant records (at Airbnb we have in excess of 10,000 keys which could be retired). It seems fairly trivial to clean as we go.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TEST PLAN

CI.

ADDITIONAL INFORMATION

  • Has associated issue:
  • Changes UI
  • Requires DB Migration.
  • Confirm DB Migration upgrade and downgrade tested.
  • Introduces new feature or API
  • Removes existing feature or API

@codecov-commenter
Copy link

codecov-commenter commented Jul 10, 2020

Codecov Report

Merging #10279 into master will decrease coverage by 0.05%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #10279      +/-   ##
==========================================
- Coverage   65.47%   65.42%   -0.06%     
==========================================
  Files         598      598              
  Lines       31992    32002      +10     
  Branches     3234     3234              
==========================================
- Hits        20946    20936      -10     
- Misses      10865    10885      +20     
  Partials      181      181              
Flag Coverage Δ
#javascript 59.39% <ø> (-0.01%) ⬇️
#python 69.61% <100.00%> (-0.09%) ⬇️
Impacted Files Coverage Δ
superset/views/core.py 74.39% <100.00%> (-0.23%) ⬇️
superset/db_engine_specs/mysql.py 78.72% <0.00%> (-12.77%) ⬇️
superset/models/core.py 85.83% <0.00%> (-0.84%) ⬇️
...set-frontend/src/views/datasetList/DatasetList.tsx 68.25% <0.00%> (-0.80%) ⬇️
superset/connectors/sqla/models.py 88.80% <0.00%> (-0.15%) ⬇️
superset/errors.py 100.00% <0.00%> (ø)
superset/viz_sip38.py 0.00% <0.00%> (ø)
superset/sql_parse.py 99.31% <0.00%> (+<0.01%) ⬆️
superset/db_engine_specs/postgres.py 100.00% <0.00%> (+2.63%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update af864e9...5f94beb. Read the comment docs.

Copy link
Member

@villebro villebro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Non-blocking recommendation: to make life easier for other developers, a small comment explaining why this task should be explicitly forgotten in this context could be helpful.

@john-bodley john-bodley force-pushed the john-bodley--celery-results-forget branch from 5f94beb to 863a5d8 Compare July 10, 2020 16:15
@john-bodley john-bodley merged commit 4e4ccd4 into apache:master Jul 10, 2020
@john-bodley john-bodley deleted the john-bodley--celery-results-forget branch July 10, 2020 16:54
auxten pushed a commit to auxten/incubator-superset that referenced this pull request Nov 20, 2020
Co-authored-by: John Bodley <john.bodley@airbnb.com>
@mistercrunch mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 0.37.0 labels Mar 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels size/XS 🚢 0.37.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants