Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: create fk model in benchmark script #15804

Merged
merged 1 commit into from Jul 20, 2021

Conversation

betodealmeida
Copy link
Member

@betodealmeida betodealmeida commented Jul 20, 2021

SUMMARY

When identifying models, the benchmark migration script should also take into consideration models that are referenced in foreign keys.

For example, running the script on 301362411006_add_execution_id_to_report_execution_.py would identify a single model, for the report_execution_log table. But that table has a foreign key to report_schedule.id. On a particular run the script was failing because report_schedule was empty.

I changed the script to also populate the models referenced in foreign keys. Ideally we'd need only 1 entity for each of those models, but for now the script generates multiple entities of them as well. I'll clean it up in a follow up PR.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

N/A

TESTING INSTRUCTIONS

Make sure you have no reports scheduled. Run:

% python scripts/benchmark_migration.py superset/migrations/versions/301362411006_add_execution_id_to_report_execution_.py

It should work:

...
Identifying models used in the migration:
- ab_user (1 rows in table ab_user)
- dashboards (12 rows in table dashboards)
- slices (114 rows in table slices)
- dbs (2 rows in table dbs)
- report_schedule (0 rows in table report_schedule)
- report_execution_log (0 rows in table report_execution_log)
...
Results:

Current: 0.20 s
10+: 0.37 s
100+: 0.19 s
1000+: 0.20 s

ADDITIONAL INFORMATION

  • Has associated issue:
  • Changes UI
  • Includes DB Migration (follow approval process in SIP-59)
    • Migration is atomic, supports rollback & is backwards-compatible
    • Confirm DB migration upgrade and downgrade tested
    • Runtime estimates and downtime expectations provided
  • Introduces new feature or API
  • Removes existing feature or API

@codecov
Copy link

codecov bot commented Jul 20, 2021

Codecov Report

Merging #15804 (83ed3d2) into master (cbd3780) will decrease coverage by 0.00%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #15804      +/-   ##
==========================================
- Coverage   76.91%   76.90%   -0.01%     
==========================================
  Files         983      983              
  Lines       51643    51643              
  Branches     6991     6991              
==========================================
- Hits        39720    39717       -3     
- Misses      11699    11702       +3     
  Partials      224      224              
Flag Coverage Δ
javascript 71.81% <ø> (-0.02%) ⬇️
mysql 81.54% <ø> (ø)
postgres 81.57% <ø> (ø)
python 81.66% <ø> (ø)
sqlite 81.18% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
superset-frontend/src/components/Select/Select.tsx 72.25% <0.00%> (-1.74%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cbd3780...83ed3d2. Read the comment docs.

@betodealmeida betodealmeida merged commit 5cc95bb into master Jul 20, 2021
lyndsiWilliams pushed a commit to preset-io/superset that referenced this pull request Jul 26, 2021
eschutho pushed a commit that referenced this pull request Jul 28, 2021
#15856)

* style: Rough draft of email report modal (#15666)

* clears errors when closing out of modal (#15623)

* add test + fix types

* fix lint errors

* Building ReportModal component

* Continued ReportModal creation

* Visual details updated

* CronError style

* Very basic testing added

Co-authored-by: AAfghahi <48933336+AAfghahi@users.noreply.github.com>

* feat: toggle the ReportModal Icon based on user permissions (#15780)

* style: Rough draft of email report modal (#15666)

* clears errors when closing out of modal (#15623)

* fix: avoid fetching favorite status for anonymous user (#15590)

* avoid fetching favorite status for anonymous user

* add test + fix types

* fix lint errors

* Building ReportModal component

* Continued ReportModal creation

* Visual details updated

* CronError style

* Very basic testing added

Co-authored-by: AAfghahi <48933336+AAfghahi@users.noreply.github.com>
Co-authored-by: aspedrosa <aspedrosa@ua.pt>

* first draft

* added testing props

* cleaned up rebase

* changed name and type

Co-authored-by: Lyndsi Kay Williams <55605634+lyndsiWilliams@users.noreply.github.com>
Co-authored-by: aspedrosa <aspedrosa@ua.pt>

* feat: Begin incorporation of email report modal to Charts (#15793)

* Add email report modal to Charts

* Fix px themes

* feat: fetch UI specific reports  (#15785)

* style: Rough draft of email report modal (#15666)

* clears errors when closing out of modal (#15623)

* fix: avoid fetching favorite status for anonymous user (#15590)

* avoid fetching favorite status for anonymous user

* add test + fix types

* fix lint errors

* Building ReportModal component

* Continued ReportModal creation

* Visual details updated

* CronError style

* Very basic testing added

Co-authored-by: AAfghahi <48933336+AAfghahi@users.noreply.github.com>
Co-authored-by: aspedrosa <aspedrosa@ua.pt>

* first draft

* created Api call

* added click logic

* made the fetch report into a action/reducer

* abstracted report action

* revision

* added reportState to reducer

* reports conditions

* revisions

* revisions

Co-authored-by: Lyndsi Kay Williams <55605634+lyndsiWilliams@users.noreply.github.com>
Co-authored-by: aspedrosa <aspedrosa@ua.pt>

* fix: Edit physical dataset from the Edit Dataset modal (#15770)

* Remove unnecessary onChange

* Remove confliciting onChange

* Revert unnecessary change

* Enhance and fix tests

* feat: add show columns to Reports model (#15712)

* added logic for creation_method

* revisions

* added index

* Update superset/migrations/versions/3317e9248280_add_creation_method_to_reports_model.py

* filters

* search columns updated

* fix: margin right on warning icon to 8px (#15715)

* changed margin right on warning icon to 8px

* fixed to grid units from pixels

* feat: adding Progress Bar to Benchmark script (#15719)

* rough draft of benchmark script

* revisions

* revisions

* rough draft of benchmark script

* revisions

* Update requirements/development.in

Co-authored-by: Beto Dealmeida <roberto@dealmeida.net>

* Update superset/utils/mock_data.py

Co-authored-by: Beto Dealmeida <roberto@dealmeida.net>

* more revisions

Co-authored-by: Beto Dealmeida <roberto@dealmeida.net>

* fix: migration downgrade references wrong column (#15791)

* fix(dashboard): Add resize handles to right and bottom of component (#15778)

* fix(dashboard): Add resize handles to right and bottom of component

* Fix test

* chore: Add tags to the new viz gallery (#15734)

* chore: add tags to gallery

* fix UT

* fix lint

* redesign tags filter

* chore: change to Highly-used and fix some css

* fix UT

Co-authored-by: stephenLYZ <750188453@qq.com>

* chore: bump 0.17.70 (#15795)

* docs: Adding Sunbird to users list (#15794)

* chore: Improves the Select component UI/UX - iteration 4 (#15480)

* Add z-index only on maximize (#15800)

* chore: remove `retry` dependency in favor of `backoff` (#15788)

* chore: remove retry dep in favor of backoff

* Fix lint

* fix: create fk model in benchmark script (#15804)

* style: Rough draft of email report modal (#15666)

* clears errors when closing out of modal (#15623)

* fix: avoid fetching favorite status for anonymous user (#15590)

* avoid fetching favorite status for anonymous user

* add test + fix types

* fix lint errors

* Building ReportModal component

* Continued ReportModal creation

* Visual details updated

* CronError style

* Very basic testing added

Co-authored-by: AAfghahi <48933336+AAfghahi@users.noreply.github.com>
Co-authored-by: aspedrosa <aspedrosa@ua.pt>

* Working on onSave functionality

* style: Rough draft of email report modal (#15666)

* clears errors when closing out of modal (#15623)

* add test + fix types

* fix lint errors

* Building ReportModal component

* Continued ReportModal creation

* Visual details updated

* CronError style

* Very basic testing added

Co-authored-by: AAfghahi <48933336+AAfghahi@users.noreply.github.com>

* feat: toggle the ReportModal Icon based on user permissions (#15780)

* style: Rough draft of email report modal (#15666)

* clears errors when closing out of modal (#15623)

* fix: avoid fetching favorite status for anonymous user (#15590)

* avoid fetching favorite status for anonymous user

* add test + fix types

* fix lint errors

* Building ReportModal component

* Continued ReportModal creation

* Visual details updated

* CronError style

* Very basic testing added

Co-authored-by: AAfghahi <48933336+AAfghahi@users.noreply.github.com>
Co-authored-by: aspedrosa <aspedrosa@ua.pt>

* first draft

* added testing props

* cleaned up rebase

* changed name and type

Co-authored-by: Lyndsi Kay Williams <55605634+lyndsiWilliams@users.noreply.github.com>
Co-authored-by: aspedrosa <aspedrosa@ua.pt>

* feat: fetch UI specific reports  (#15785)

* style: Rough draft of email report modal (#15666)

* clears errors when closing out of modal (#15623)

* fix: avoid fetching favorite status for anonymous user (#15590)

* avoid fetching favorite status for anonymous user

* add test + fix types

* fix lint errors

* Building ReportModal component

* Continued ReportModal creation

* Visual details updated

* CronError style

* Very basic testing added

Co-authored-by: AAfghahi <48933336+AAfghahi@users.noreply.github.com>
Co-authored-by: aspedrosa <aspedrosa@ua.pt>

* first draft

* created Api call

* added click logic

* made the fetch report into a action/reducer

* abstracted report action

* revision

* added reportState to reducer

* reports conditions

* revisions

* revisions

Co-authored-by: Lyndsi Kay Williams <55605634+lyndsiWilliams@users.noreply.github.com>
Co-authored-by: aspedrosa <aspedrosa@ua.pt>

* Dashboard onSave progress

* More onSave/create progress

* Dashboard POST working!

* Dashboard POST relocated to redux

* POST now also working in Charts

* making linter happy

* linter again

Co-authored-by: AAfghahi <48933336+AAfghahi@users.noreply.github.com>
Co-authored-by: aspedrosa <aspedrosa@ua.pt>
Co-authored-by: Geido <60598000+geido@users.noreply.github.com>
Co-authored-by: stellalc7 <stellalc7@gmail.com>
Co-authored-by: Beto Dealmeida <roberto@dealmeida.net>
Co-authored-by: Kamil Gabryjelski <kamil.gabryjelski@gmail.com>
Co-authored-by: Yongjie Zhao <yongjie.zhao@gmail.com>
Co-authored-by: stephenLYZ <750188453@qq.com>
Co-authored-by: Kumar <kumarks1122@gmail.com>
Co-authored-by: Michael S. Molina <70410625+michael-s-molina@users.noreply.github.com>
eschutho pushed a commit that referenced this pull request Jul 30, 2021
#15856)

* style: Rough draft of email report modal (#15666)

* clears errors when closing out of modal (#15623)

* add test + fix types

* fix lint errors

* Building ReportModal component

* Continued ReportModal creation

* Visual details updated

* CronError style

* Very basic testing added

Co-authored-by: AAfghahi <48933336+AAfghahi@users.noreply.github.com>

* feat: toggle the ReportModal Icon based on user permissions (#15780)

* style: Rough draft of email report modal (#15666)

* clears errors when closing out of modal (#15623)

* fix: avoid fetching favorite status for anonymous user (#15590)

* avoid fetching favorite status for anonymous user

* add test + fix types

* fix lint errors

* Building ReportModal component

* Continued ReportModal creation

* Visual details updated

* CronError style

* Very basic testing added

Co-authored-by: AAfghahi <48933336+AAfghahi@users.noreply.github.com>
Co-authored-by: aspedrosa <aspedrosa@ua.pt>

* first draft

* added testing props

* cleaned up rebase

* changed name and type

Co-authored-by: Lyndsi Kay Williams <55605634+lyndsiWilliams@users.noreply.github.com>
Co-authored-by: aspedrosa <aspedrosa@ua.pt>

* feat: Begin incorporation of email report modal to Charts (#15793)

* Add email report modal to Charts

* Fix px themes

* feat: fetch UI specific reports  (#15785)

* style: Rough draft of email report modal (#15666)

* clears errors when closing out of modal (#15623)

* fix: avoid fetching favorite status for anonymous user (#15590)

* avoid fetching favorite status for anonymous user

* add test + fix types

* fix lint errors

* Building ReportModal component

* Continued ReportModal creation

* Visual details updated

* CronError style

* Very basic testing added

Co-authored-by: AAfghahi <48933336+AAfghahi@users.noreply.github.com>
Co-authored-by: aspedrosa <aspedrosa@ua.pt>

* first draft

* created Api call

* added click logic

* made the fetch report into a action/reducer

* abstracted report action

* revision

* added reportState to reducer

* reports conditions

* revisions

* revisions

Co-authored-by: Lyndsi Kay Williams <55605634+lyndsiWilliams@users.noreply.github.com>
Co-authored-by: aspedrosa <aspedrosa@ua.pt>

* fix: Edit physical dataset from the Edit Dataset modal (#15770)

* Remove unnecessary onChange

* Remove confliciting onChange

* Revert unnecessary change

* Enhance and fix tests

* feat: add show columns to Reports model (#15712)

* added logic for creation_method

* revisions

* added index

* Update superset/migrations/versions/3317e9248280_add_creation_method_to_reports_model.py

* filters

* search columns updated

* fix: margin right on warning icon to 8px (#15715)

* changed margin right on warning icon to 8px

* fixed to grid units from pixels

* feat: adding Progress Bar to Benchmark script (#15719)

* rough draft of benchmark script

* revisions

* revisions

* rough draft of benchmark script

* revisions

* Update requirements/development.in

Co-authored-by: Beto Dealmeida <roberto@dealmeida.net>

* Update superset/utils/mock_data.py

Co-authored-by: Beto Dealmeida <roberto@dealmeida.net>

* more revisions

Co-authored-by: Beto Dealmeida <roberto@dealmeida.net>

* fix: migration downgrade references wrong column (#15791)

* fix(dashboard): Add resize handles to right and bottom of component (#15778)

* fix(dashboard): Add resize handles to right and bottom of component

* Fix test

* chore: Add tags to the new viz gallery (#15734)

* chore: add tags to gallery

* fix UT

* fix lint

* redesign tags filter

* chore: change to Highly-used and fix some css

* fix UT

Co-authored-by: stephenLYZ <750188453@qq.com>

* chore: bump 0.17.70 (#15795)

* docs: Adding Sunbird to users list (#15794)

* chore: Improves the Select component UI/UX - iteration 4 (#15480)

* Add z-index only on maximize (#15800)

* chore: remove `retry` dependency in favor of `backoff` (#15788)

* chore: remove retry dep in favor of backoff

* Fix lint

* fix: create fk model in benchmark script (#15804)

* style: Rough draft of email report modal (#15666)

* clears errors when closing out of modal (#15623)

* fix: avoid fetching favorite status for anonymous user (#15590)

* avoid fetching favorite status for anonymous user

* add test + fix types

* fix lint errors

* Building ReportModal component

* Continued ReportModal creation

* Visual details updated

* CronError style

* Very basic testing added

Co-authored-by: AAfghahi <48933336+AAfghahi@users.noreply.github.com>
Co-authored-by: aspedrosa <aspedrosa@ua.pt>

* Working on onSave functionality

* style: Rough draft of email report modal (#15666)

* clears errors when closing out of modal (#15623)

* add test + fix types

* fix lint errors

* Building ReportModal component

* Continued ReportModal creation

* Visual details updated

* CronError style

* Very basic testing added

Co-authored-by: AAfghahi <48933336+AAfghahi@users.noreply.github.com>

* feat: toggle the ReportModal Icon based on user permissions (#15780)

* style: Rough draft of email report modal (#15666)

* clears errors when closing out of modal (#15623)

* fix: avoid fetching favorite status for anonymous user (#15590)

* avoid fetching favorite status for anonymous user

* add test + fix types

* fix lint errors

* Building ReportModal component

* Continued ReportModal creation

* Visual details updated

* CronError style

* Very basic testing added

Co-authored-by: AAfghahi <48933336+AAfghahi@users.noreply.github.com>
Co-authored-by: aspedrosa <aspedrosa@ua.pt>

* first draft

* added testing props

* cleaned up rebase

* changed name and type

Co-authored-by: Lyndsi Kay Williams <55605634+lyndsiWilliams@users.noreply.github.com>
Co-authored-by: aspedrosa <aspedrosa@ua.pt>

* feat: fetch UI specific reports  (#15785)

* style: Rough draft of email report modal (#15666)

* clears errors when closing out of modal (#15623)

* fix: avoid fetching favorite status for anonymous user (#15590)

* avoid fetching favorite status for anonymous user

* add test + fix types

* fix lint errors

* Building ReportModal component

* Continued ReportModal creation

* Visual details updated

* CronError style

* Very basic testing added

Co-authored-by: AAfghahi <48933336+AAfghahi@users.noreply.github.com>
Co-authored-by: aspedrosa <aspedrosa@ua.pt>

* first draft

* created Api call

* added click logic

* made the fetch report into a action/reducer

* abstracted report action

* revision

* added reportState to reducer

* reports conditions

* revisions

* revisions

Co-authored-by: Lyndsi Kay Williams <55605634+lyndsiWilliams@users.noreply.github.com>
Co-authored-by: aspedrosa <aspedrosa@ua.pt>

* Dashboard onSave progress

* More onSave/create progress

* Dashboard POST working!

* Dashboard POST relocated to redux

* POST now also working in Charts

* making linter happy

* linter again

Co-authored-by: AAfghahi <48933336+AAfghahi@users.noreply.github.com>
Co-authored-by: aspedrosa <aspedrosa@ua.pt>
Co-authored-by: Geido <60598000+geido@users.noreply.github.com>
Co-authored-by: stellalc7 <stellalc7@gmail.com>
Co-authored-by: Beto Dealmeida <roberto@dealmeida.net>
Co-authored-by: Kamil Gabryjelski <kamil.gabryjelski@gmail.com>
Co-authored-by: Yongjie Zhao <yongjie.zhao@gmail.com>
Co-authored-by: stephenLYZ <750188453@qq.com>
Co-authored-by: Kumar <kumarks1122@gmail.com>
Co-authored-by: Michael S. Molina <70410625+michael-s-molina@users.noreply.github.com>
cccs-RyanS pushed a commit to CybercentreCanada/superset that referenced this pull request Dec 17, 2021
QAlexBall pushed a commit to QAlexBall/superset that referenced this pull request Dec 29, 2021
cccs-rc pushed a commit to CybercentreCanada/superset that referenced this pull request Mar 6, 2024
@mistercrunch mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 1.3.0 labels Mar 13, 2024
@mistercrunch mistercrunch deleted the fix_benchmark_script_fks branch March 26, 2024 16:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels preset:2021.29 🚢 1.3.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants