Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve test coverage #2900

Closed
MeltyBot opened this issue Sep 30, 2021 · 5 comments
Closed

Improve test coverage #2900

MeltyBot opened this issue Sep 30, 2021 · 5 comments
Labels
discussion kind/Tech Debt migrated from gitlab valuestream/Meltano

Comments

@MeltyBot
Copy link
Contributor

MeltyBot commented Sep 30, 2021 •
edited
Loading

Migrated from GitLab: https://gitlab.com/meltano/meltano/-/issues/2959

Originally created by @pandemicsyn on 2021-09-30 15:18:18

This is a bit of a meta/informational issue - but while helping with https://gitlab.com/meltano/meltano/-/merge_requests/2349 (small bug in controllers/orchestrations) realized we have some bigger gaps in our test coverage. So I ran a "quick" coverage report. Here's the raw result (but feel free to skip ahead, I include two top 20 reports):

Raw coverage report

---------- coverage: platform darwin, python 3.8.11-final-0 ----------
Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
src/meltano/__init__.py                                  1      0   100%
src/meltano/api/__init__.py                              0      0   100%
src/meltano/api/__main__.py                              0      0   100%
src/meltano/api/api_blueprint.py                        23      1    96%
src/meltano/api/app.py                                 104      8    92%
src/meltano/api/config.py                               37      0   100%
src/meltano/api/controllers/__init__.py                  0      0   100%
src/meltano/api/controllers/dashboards.py               71     27    62%
src/meltano/api/controllers/dashboards_helper.py        13      7    46%
src/meltano/api/controllers/embeds.py                   25      1    96%
src/meltano/api/controllers/embeds_helper.py            59     29    51%
src/meltano/api/controllers/errors.py                    3      1    67%
src/meltano/api/controllers/orchestrations.py          271    131    52%
src/meltano/api/controllers/plugins.py                  80     38    52%
src/meltano/api/controllers/reports.py                  53     22    58%
src/meltano/api/controllers/reports_helper.py           25     18    28%
src/meltano/api/controllers/repos.py                   107     37    65%
src/meltano/api/controllers/repos_helper.py             11      0   100%
src/meltano/api/controllers/root.py                     73     15    79%
src/meltano/api/controllers/settings.py                133     48    64%
src/meltano/api/controllers/settings_helper.py          33     23    30%
src/meltano/api/controllers/sql.py                      67     19    72%
src/meltano/api/controllers/sql_helper.py               78     23    71%
src/meltano/api/controllers/upload_helper.py            32     18    44%
src/meltano/api/controllers/utils.py                     8      1    88%
src/meltano/api/events/__init__.py                       3      3     0%
src/meltano/api/events/notification_events.py           50     50     0%
src/meltano/api/executor/__init__.py                    31      8    74%
src/meltano/api/headers.py                               2      0   100%
src/meltano/api/json.py                                 58      7    88%
src/meltano/api/mail.py                                 27     13    52%
src/meltano/api/models/__init__.py                       2      0   100%
src/meltano/api/models/embed_token.py                   17      0   100%
src/meltano/api/models/oauth.py                         15      0   100%
src/meltano/api/models/security.py                      52     10    81%
src/meltano/api/models/subscription.py                  22      3    86%
src/meltano/api/profiler.py                             30     30     0%
src/meltano/api/security/__init__.py                    22      0   100%
src/meltano/api/security/auth.py                        82      7    91%
src/meltano/api/security/forms.py                       22      5    77%
src/meltano/api/security/identity.py                    43      6    86%
src/meltano/api/security/oauth.py                       65     15    77%
src/meltano/api/security/resource_filter.py             36      1    97%
src/meltano/api/signals.py                               7      1    86%
src/meltano/api/url_converters.py                        7      0   100%
src/meltano/api/workers/__init__.py                      3      0   100%
src/meltano/api/workers/api_worker.py                   22      7    68%
src/meltano/api/workers/meltano_compiler_worker.py      35     10    71%
src/meltano/api/workers/ui_available_worker.py          32      3    91%
src/meltano/api/wsgi.py                                 27     27     0%
src/meltano/cli/__init__.py                             24      2    92%
src/meltano/cli/__main__.py                              2      2     0%
src/meltano/cli/add.py                                  57      2    96%
src/meltano/cli/cli.py                                  35      5    86%
src/meltano/cli/config.py                              191     85    55%
src/meltano/cli/discovery.py                            29      0   100%
src/meltano/cli/elt.py                                 170      6    96%
src/meltano/cli/initialize.py                           37      5    86%
src/meltano/cli/install.py                              34      4    88%
src/meltano/cli/invoke.py                               74      7    91%
src/meltano/cli/model.py                                47     30    36%
src/meltano/cli/params.py                               37      2    95%
src/meltano/cli/remove.py                               43     25    42%
src/meltano/cli/repl.py                                 13      8    38%
src/meltano/cli/schedule.py                             80     24    70%
src/meltano/cli/schema.py                               18      3    83%
src/meltano/cli/select.py                               70     32    54%
src/meltano/cli/ui.py                                   99     23    77%
src/meltano/cli/upgrade.py                              66      0   100%
src/meltano/cli/user.py                                 38     20    47%
src/meltano/cli/utils.py                               232     39    83%
src/meltano/core/__init__.py                             0      0   100%
src/meltano/core/behavior/__init__.py                    1      0   100%
src/meltano/core/behavior/canonical.py                  97      2    98%
src/meltano/core/behavior/hookable.py                   49      5    90%
src/meltano/core/behavior/name_eq.py                     5      1    80%
src/meltano/core/behavior/versioned.py                  22      6    73%
src/meltano/core/behavior/visitor.py                    11      0   100%
src/meltano/core/bundle/__init__.py                      6      0   100%
src/meltano/core/compiler/__init__.py                    0      0   100%
src/meltano/core/compiler/m5oc_file.py                  10     10     0%
src/meltano/core/compiler/project_compiler.py           33      7    79%
src/meltano/core/config_service.py                      39      1    97%
src/meltano/core/connection_service.py                  33      0   100%
src/meltano/core/db.py                                  61     18    70%
src/meltano/core/elt_context.py                        133      0   100%
src/meltano/core/error.py                              107     55    49%
src/meltano/core/extract_utils.py                       66     66     0%
src/meltano/core/extractor.py                            5      5     0%
src/meltano/core/job/__init__.py                         2      0   100%
src/meltano/core/job/finder.py                          30      3    90%
src/meltano/core/job/job.py                            153      7    95%
src/meltano/core/job/stale_job_failer.py                20      1    95%
src/meltano/core/logging/__init__.py                     3      0   100%
src/meltano/core/logging/job_logging_service.py         70     16    77%
src/meltano/core/logging/output_logger.py              137      6    96%
src/meltano/core/logging/utils.py                       51      4    92%
src/meltano/core/m5o/__init__.py                         0      0   100%
src/meltano/core/m5o/dashboards_service.py             107     33    69%
src/meltano/core/m5o/m5o_collection_parser.py           26      3    88%
src/meltano/core/m5o/m5o_file_parser.py                277     26    91%
src/meltano/core/m5o/m5oc_file.py                       37     11    70%
src/meltano/core/m5o/reports_service.py                 76     20    74%
src/meltano/core/meltano_file.py                        23      0   100%
src/meltano/core/meltano_invoker.py                     29      0   100%
src/meltano/core/migration_service.py                   67      4    94%
src/meltano/core/models.py                               4      0   100%
src/meltano/core/plugin/__init__.py                      1      0   100%
src/meltano/core/plugin/airflow.py                      72      4    94%
src/meltano/core/plugin/base.py                        197      4    98%
src/meltano/core/plugin/command.py                      39      1    97%
src/meltano/core/plugin/config_service.py               26      0   100%
src/meltano/core/plugin/connection.py                    0      0   100%
src/meltano/core/plugin/dashboard.py                   129     18    86%
src/meltano/core/plugin/dbt/__init__.py                  1      0   100%
src/meltano/core/plugin/dbt/base.py                     48      9    81%
src/meltano/core/plugin/error.py                        25      1    96%
src/meltano/core/plugin/factory.py                      12      0   100%
src/meltano/core/plugin/file.py                         71      1    99%
src/meltano/core/plugin/meltano_file.py                 16      1    94%
src/meltano/core/plugin/model/__init__.py                2      0   100%
src/meltano/core/plugin/model/base.py                    3      0   100%
src/meltano/core/plugin/model/package.py                16      0   100%
src/meltano/core/plugin/project_plugin.py              111      2    98%
src/meltano/core/plugin/settings_service.py             47      1    98%
src/meltano/core/plugin/singer/__init__.py               6      0   100%
src/meltano/core/plugin/singer/base.py                  41      0   100%
src/meltano/core/plugin/singer/catalog.py              280      9    97%
src/meltano/core/plugin/singer/tap.py                  222     25    89%
src/meltano/core/plugin/singer/target.py                60      8    87%
src/meltano/core/plugin/utility.py                       3      0   100%
src/meltano/core/plugin_discovery_service.py           172      6    97%
src/meltano/core/plugin_install_service.py             127     23    82%
src/meltano/core/plugin_invoker.py                     152      3    98%
src/meltano/core/plugin_location_remove.py              80      8    90%
src/meltano/core/plugin_remove_service.py               28      0   100%
src/meltano/core/plugin_test_service.py                 47      1    98%
src/meltano/core/project.py                            170      9    95%
src/meltano/core/project_add_service.py                 38      0   100%
src/meltano/core/project_init_service.py                68      5    93%
src/meltano/core/project_plugins_service.py            109     10    91%
src/meltano/core/project_settings_service.py            53      2    96%
src/meltano/core/runner/__init__.py                      7      1    86%
src/meltano/core/runner/dbt.py                          40      4    90%
src/meltano/core/runner/singer.py                      113     13    88%
src/meltano/core/schedule.py                            23      2    91%
src/meltano/core/schedule_service.py                    81      8    90%
src/meltano/core/select_service.py                      49     13    73%
src/meltano/core/setting.py                             17      2    88%
src/meltano/core/setting_definition.py                 122      6    95%
src/meltano/core/settings_service.py                   216     12    94%
src/meltano/core/settings_store.py                     439     16    96%
src/meltano/core/sql/__init__.py                         0      0   100%
src/meltano/core/sql/aggregate.py                       39     17    56%
src/meltano/core/sql/analysis_helper.py                 43     14    67%
src/meltano/core/sql/base.py                           676     65    90%
src/meltano/core/sql/date.py                            22     15    32%
src/meltano/core/sql/design_helper.py                  123     22    82%
src/meltano/core/sql/filter.py                           5      0   100%
src/meltano/core/sql/sql_utils.py                        7      0   100%
src/meltano/core/sql/substitution.py                    54     37    31%
src/meltano/core/sql/timeframe.py                       13      5    62%
src/meltano/core/sqlalchemy.py                          41      5    88%
src/meltano/core/tracking/__init__.py                    1      0   100%
src/meltano/core/tracking/ga_tracker.py                 97     31    68%
src/meltano/core/transform_add_service.py               59      5    92%
src/meltano/core/upgrade_service.py                     99     31    69%
src/meltano/core/utils/__init__.py                     204     19    91%
src/meltano/core/utils/pidfile.py                       42     23    45%
src/meltano/core/venv_service.py                       125     13    90%
src/meltano/migrations/__init__.py                       7      0   100%
src/meltano/migrations/env.py                           23     23     0%
src/meltano/models.py                                    2      2     0%
src/meltano/oauth/__init__.py                            1      0   100%
src/meltano/oauth/app.py                                31      5    84%
src/meltano/oauth/config.py                              6      0   100%
src/meltano/oauth/providers.py                          29      9    69%
------------------------------------------------------------------------
TOTAL                                                10533   1876    82%

Top 20

That above is large list to grok, so here's two top 20's. The first is a top 20 files sorted by coverage percentage:

(meltano-IZTQDpuZ-py3.8) ➜  /tmp grep src test.txt | sort -k 4 | head -n 20
src/meltano/api/events/__init__.py                       3      3     0%
src/meltano/api/events/notification_events.py           50     50     0%
src/meltano/api/profiler.py                             30     30     0%
src/meltano/api/wsgi.py                                 27     27     0%
src/meltano/cli/__main__.py                              2      2     0%
src/meltano/core/compiler/m5oc_file.py                  10     10     0%
src/meltano/core/extract_utils.py                       66     66     0%
src/meltano/core/extractor.py                            5      5     0%
src/meltano/migrations/env.py                           23     23     0%
src/meltano/models.py                                    2      2     0%
src/meltano/api/controllers/reports_helper.py           25     18    28%
src/meltano/api/controllers/settings_helper.py          33     23    30%
src/meltano/core/sql/substitution.py                    54     37    31%
src/meltano/core/sql/date.py                            22     15    32%
src/meltano/cli/model.py                                47     30    36%
src/meltano/cli/repl.py                                 13      8    38%
src/meltano/cli/remove.py                               43     25    42%
src/meltano/api/controllers/upload_helper.py            32     18    44%
src/meltano/core/utils/pidfile.py                       42     23    45%
src/meltano/api/controllers/dashboards_helper.py        13      7    46%

This second one is sorting by # of statements missed. I feel like that's sometimes more indicative of where you've got real gaps. Case in point, orchestrations.py is #1 on this list:

(meltano-IZTQDpuZ-py3.8) ➜  /tmp grep src test.txt | sort -k 3 -r | head -n 20
src/meltano/api/controllers/orchestrations.py          271    131    52%
src/meltano/cli/config.py                              191     85    55%
src/meltano/core/extract_utils.py                       66     66     0%
src/meltano/core/sql/base.py                           676     65    90%
src/meltano/core/error.py                              107     55    49%
src/meltano/api/events/notification_events.py           50     50     0%
src/meltano/api/controllers/settings.py                133     48    64%
src/meltano/cli/utils.py                               232     39    83%
src/meltano/api/controllers/plugins.py                  80     38    52%
src/meltano/api/controllers/repos.py                   107     37    65%
src/meltano/core/sql/substitution.py                    54     37    31%
src/meltano/core/m5o/dashboards_service.py             107     33    69%
src/meltano/cli/select.py                               70     32    54%
src/meltano/core/upgrade_service.py                     99     31    69%
src/meltano/core/tracking/ga_tracker.py                 97     31    68%
src/meltano/cli/model.py                                47     30    36%
src/meltano/api/profiler.py                             30     30     0%
src/meltano/api/controllers/embeds_helper.py            59     29    51%
src/meltano/api/controllers/dashboards.py               71     27    62%
src/meltano/api/wsgi.py                                 27     27     0%

Thoughts

We've got some gaps (quite a few in controllers it seems like) but its nothing terrible. We can improve coverage for some of these as we're working on issues in adjacent areas, as long as we're diligent and look out for them.

We could also break out some small explicit issues to sprinkle throughout a few dev cycles (i.e. improve test coverage for fileX style issues). @aaronsteers @tayloramurphy I'm not sure how you'd prefer to tackle tech debt like this, but one thing I've also seen at gigs in the past is having tech debt/clean up focused dev days every so often.
@MeltyBot
Copy link
Contributor Author

View 5 previous comments from the original issue on GitLab

@stale
Copy link

stale bot commented Apr 27, 2023

This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.

@stale stale bot added the stale label Apr 27, 2023
@tayloramurphy
Copy link
Collaborator

Still seems relevant @WillDaSilva ?

@stale stale bot removed the stale label May 10, 2023
@WillDaSilva
Copy link
Member

@tayloramurphy The issue details are stale, but the problem remains relevant. I'll refresh this issue after the Meltano 3.0 release.

@edgarrmondragon
Copy link
Collaborator

Coverage is now in a much better situation, with ~92% at the moment and constantly improving in PRs. I think we can close this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
discussion kind/Tech Debt migrated from gitlab valuestream/Meltano
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@tayloramurphy @WillDaSilva @edgarrmondragon @MeltyBot