Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: use nullpool even for user lookup in the celery #10938

Merged
merged 2 commits into from
Sep 21, 2020

Conversation

bkyryliuk
Copy link
Member

@bkyryliuk bkyryliuk commented Sep 17, 2020

SUMMARY

Follow up to the #10819

Security manager still relies on the default superset db pool instead of nullpool for celery.

TEST PLAN

[x] existing unit tests
[x] local & staging env

Error msg:

  File "/srv/yaps/mounts/superset/superset_build/eb36af9819e03e516ffc7d5b9aeabbf6cb067fe5-1280802881dce54672c405f194757e5f/celery.runfiles/__main__/thirdparty/superset_python3/superset-cpython-38/lib/superset/tasks/schedules.py", line 204, in create_webdriver
    return WebDriverProxy(driver_type=config["WEBDRIVER_TYPE"]).auth(get_reports_user())
  File "/srv/yaps/mounts/superset/superset_build/eb36af9819e03e516ffc7d5b9aeabbf6cb067fe5-1280802881dce54672c405f194757e5f/celery.runfiles/__main__/thirdparty/superset_python3/superset-cpython-38/lib/superset/tasks/schedules.py", line 208, in get_reports_user
    return security_manager.find_user(config["EMAIL_REPORTS_USER"])
  File "/srv/yaps/mounts/superset/superset_build/eb36af9819e03e516ffc7d5b9aeabbf6cb067fe5-1280802881dce54672c405f194757e5f/celery.runfiles/__main__/thirdparty/superset_python3/pip/flask-appbuilder/flask-appbuilder-cpython-38/lib/flask_appbuilder/security/sqla/manager.py", line 156, in find_user
    self.get_session.query(self.user_model)
  File "/srv/yaps/mounts/superset/superset_build/eb36af9819e03e516ffc7d5b9aeabbf6cb067fe5-1280802881dce54672c405f194757e5f/celery.runfiles/__main__/pip/SQLAlchemy/SQLAlchemy-cpython-38/lib/sqlalchemy/orm/query.py", line 3432, in one_or_none
    ret = list(self)
  File "/srv/yaps/mounts/superset/superset_build/eb36af9819e03e516ffc7d5b9aeabbf6cb067fe5-1280802881dce54672c405f194757e5f/celery.runfiles/__main__/pip/SQLAlchemy/SQLAlchemy-cpython-38/lib/sqlalchemy/orm/loading.py", line 100, in instances
    cursor.close()
  File "/srv/yaps/mounts/superset/superset_build/eb36af9819e03e516ffc7d5b9aeabbf6cb067fe5-1280802881dce54672c405f194757e5f/celery.runfiles/__main__/pip/SQLAlchemy/SQLAlchemy-cpython-38/lib/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.raise_(
  File "/srv/yaps/mounts/superset/superset_build/eb36af9819e03e516ffc7d5b9aeabbf6cb067fe5-1280802881dce54672c405f194757e5f/celery.runfiles/__main__/pip/SQLAlchemy/SQLAlchemy-cpython-38/lib/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
  File "/srv/yaps/mounts/superset/superset_build/eb36af9819e03e516ffc7d5b9aeabbf6cb067fe5-1280802881dce54672c405f194757e5f/celery.runfiles/__main__/pip/SQLAlchemy/SQLAlchemy-cpython-38/lib/sqlalchemy/orm/loading.py", line 80, in instances
    rows = [proc(row) for row in fetch]
  File "/srv/yaps/mounts/superset/superset_build/eb36af9819e03e516ffc7d5b9aeabbf6cb067fe5-1280802881dce54672c405f194757e5f/celery.runfiles/__main__/pip/SQLAlchemy/SQLAlchemy-cpython-38/lib/sqlalchemy/orm/loading.py", line 80, in <listcomp>
    rows = [proc(row) for row in fetch]
  File "/srv/yaps/mounts/superset/superset_build/eb36af9819e03e516ffc7d5b9aeabbf6cb067fe5-1280802881dce54672c405f194757e5f/celery.runfiles/__main__/pip/SQLAlchemy/SQLAlchemy-cpython-38/lib/sqlalchemy/orm/loading.py", line 524, in _instance
    tuple([row[column] for column in pk_cols]),
  File "/srv/yaps/mounts/superset/superset_build/eb36af9819e03e516ffc7d5b9aeabbf6cb067fe5-1280802881dce54672c405f194757e5f/celery.runfiles/__main__/pip/SQLAlchemy/SQLAlchemy-cpython-38/lib/sqlalchemy/orm/loading.py", line 524, in <listcomp>
    tuple([row[column] for column in pk_cols]),
  File "/srv/yaps/mounts/superset/superset_build/eb36af9819e03e516ffc7d5b9aeabbf6cb067fe5-1280802881dce54672c405f194757e5f/celery.runfiles/__main__/pip/SQLAlchemy/SQLAlchemy-cpython-38/lib/sqlalchemy/engine/result.py", line 681, in _key_fallback
    util.raise_(
  File "/srv/yaps/mounts/superset/superset_build/eb36af9819e03e516ffc7d5b9aeabbf6cb067fe5-1280802881dce54672c405f194757e5f/celery.runfiles/__main__/pip/SQLAlchemy/SQLAlchemy-cpython-38/lib/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
sqlalchemy.exc.NoSuchColumnError: "Could not locate column in row for column 'ab_user.id'"

Copy link
Member

@villebro villebro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two minor non-blocking comments

Comment on lines +214 to +215
func.lower(security_manager.user_model.username)
== func.lower(config["EMAIL_REPORTS_USER"])
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Curious about this, what are the cases where it's needed?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah, this is from security manager implementation, wanted them to be aligned, email is case insensitive

func.lower(security_manager.user_model.username)
== func.lower(config["EMAIL_REPORTS_USER"])
)
.one_or_none()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: this is mostly personal preference, but I almost feel we should be able to assume that a reports user exists, and raise if it doesn't, i.e. call .one().

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good point, fixed

@codecov-commenter
Copy link

Codecov Report

Merging #10938 into master will decrease coverage by 0.06%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #10938      +/-   ##
==========================================
- Coverage   65.87%   65.81%   -0.07%     
==========================================
  Files         815      815              
  Lines       38333    38334       +1     
  Branches     3600     3600              
==========================================
- Hits        25252    25228      -24     
- Misses      12979    13002      +23     
- Partials      102      104       +2     
Flag Coverage Δ
#cypress 56.71% <ø> (-0.23%) ⬇️
#javascript 61.75% <ø> (ø)
#python 61.45% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
superset/tasks/schedules.py 76.87% <100.00%> (+0.06%) ⬆️
superset-frontend/src/reduxUtils.ts 70.88% <0.00%> (-8.87%) ⬇️
...ntend/src/dashboard/components/PropertiesModal.jsx 57.14% <0.00%> (-3.58%) ⬇️
superset-frontend/src/SqlLab/actions/sqlLab.js 62.60% <0.00%> (-2.57%) ⬇️
superset-frontend/src/SqlLab/reducers/sqlLab.js 39.58% <0.00%> (-1.67%) ⬇️
...perset-frontend/src/views/CRUD/chart/ChartList.tsx 82.75% <0.00%> (+0.86%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 801fb40...fa3f078. Read the comment docs.

@bkyryliuk bkyryliuk merged commit 56d0018 into apache:master Sep 21, 2020
@bkyryliuk bkyryliuk deleted the bogdan/fix_email_reports branch September 21, 2020 17:34
amitmiran137 pushed a commit to ofekisr/incubator-superset that referenced this pull request Sep 22, 2020
…l_access/dashboard_by_id_endpoints

* upstream/master: (29 commits)
  fix(presto): default unknown types to string type (apache#10753)
  feat(row-level-security): add base filter type and filter grouping (apache#10946)
  docs: add gallery screenshot & link in README (apache#10988)
  docs: add a "Gallery" page (apache#10968)
  build: add PR lint action (apache#10990)
  adding filters back that caused issues (apache#10989)
  chore: selectors refactor in SQLLab test suite (Cypress) (apache#10944)
  ESLint: Remove ts-ignore comments (apache#10933)
  style: fix checkbox color (apache#10970)
  fix: changed disabled rules in datasets module (apache#10979)
  fix: update the time filter for 'Last Year' option in explore (apache#10829)
  fix: use nullpool even for user lookup in the celery (apache#10938)
  Allow empty observations in alerting (apache#10939)
  fix: re-enabling several globally disabled lint rules (apache#10957)
  fix: setting specific exceptions common/query_context.py (apache#10942)
  Pylint disabled rule `pointless-string-statement` is not raising warining anymore - removing (apache#10975)
  fix: pylint disabled rules in dashboard/api.py (apache#10976)
  fix: removed disabled lint rule `too-many-locals` in connectors/base/models.py (apache#10958)
  ESLint: Re-enable rule no-access-state-in-setstate (apache#10870)
  fix: simply is_adhoc_metric (apache#10964)
  ...
auxten pushed a commit to auxten/incubator-superset that referenced this pull request Nov 20, 2020
* Use nullpool even for user lookup in the celery

* Address feedback

Co-authored-by: bogdan kyryliuk <bogdankyryliuk@dropbox.com>
@mistercrunch mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 0.38.0 labels Mar 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels size/M 🚢 0.38.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants