Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace connect-specific entry points with one global entry point #308

Merged

Conversation

mars-lan
Copy link
Contributor

@mars-lan mars-lan commented Aug 1, 2022

🤔 Why?

To simplify the user experience

Before:

python -m metaphor.bigquery bigquery.yml

After:

metaphor bigquery bigquery.yml

🤓 What?

  • Replace connector-specific entry points with a global "metaphor" CLI entry point
  • Update all relevant README files
  • Remove the deprecated and unused Google and Slack directory connectors

🧪 Tested?

Built a test Docker image and manually verified all connectors end-to-end.

@shortcut-integration
Copy link

@mars-lan mars-lan mentioned this pull request Aug 1, 2022
@mars-lan mars-lan force-pushed the marslan/sc-10762/replace-crawler-specific-entry-points-with branch from d1c3cb5 to 7ef7cd6 Compare August 1, 2022 18:09
@mars-lan mars-lan changed the title Replace connect-specific entry point with one global entry point Replace connect-specific entry points with one global entry point Aug 1, 2022
@github-actions
Copy link

github-actions bot commented Aug 1, 2022

Coverage

Coverage Report
FileStmtsMissCoverMissing
metaphor
   main.py13130%1–22
metaphor/airflow_plugin/lineage
   backend.py912078%12–13, 38–50, 80–81, 84–85, 88, 99–100, 126, 168, 173
metaphor/bigquery
   init.py4175%6
   extractor.py961980%6–8, 36, 39, 78–82, 104–105, 125–126, 157, 163–169, 187
   logEvent.py64691%45–46, 90–91, 98, 113
   utils.py581476%14–16, 28–29, 36–37, 46–53, 102, 123, 127
metaphor/bigquery/lineage
   init.py4175%6
   extractor.py1202381%7–9, 37, 40, 75–76, 89–90, 96–97, 102, 113, 115, 119, 153–154, 157, 164, 167–170, 178–179
metaphor/bigquery/profile
   init.py4175%6
   extractor.py1546955%9–11, 38, 41, 45, 48, 53–61, 68–90, 98–116, 124–164
metaphor/bigquery/query
   init.py4175%6
   extractor.py67987%29, 32, 64, 76, 79–80, 90–91, 100
metaphor/bigquery/usage
   init.py4175%6
   extractor.py94990%31, 35–36, 63, 66, 97, 117–118, 121
metaphor/common
   api_sink.py362044%26–29, 33–60
   cli.py6350%7–9
   event_util.py51688%55, 57, 61, 68–70
   extractor.py651774%44–51, 65–69, 94, 98–99, 102–103
   file_sink.py561377%50, 78–90, 94–95
   filter.py71199%32
   git.py28582%42–48
   query_history.py27196%31
   sink.py20385%21, 28–29
   storage.py671873%69, 91–92, 103–112, 115–131, 138–140, 151
metaphor/dbt
   init.py4175%6
   catalog_parser_v1.py753060%47–49, 63, 82, 101, 107–109, 113–146
   extractor.py52394%31, 34, 84
   manifest_parser_v3.py1161191%77–79, 152, 160, 183–184, 212–213, 262, 288
   manifest_parser_v5.py1291390%85–87, 162, 170, 196–197, 223–224, 258, 318, 338, 344
   util.py98595%52, 177–180
metaphor/dbt/cloud
   init.py4175%6
   extractor.py682859%28–29, 32–43, 48–69, 78–83, 89–94, 103, 106
metaphor/dbt/generated
   dbt_manifest_v1.py8478470%7–1307
   dbt_manifest_v2.py8488480%7–1324
metaphor/looker
   init.py4175%6
   extractor.py896428%11–13, 40, 43, 47, 71–76, 79–102, 107–143, 150–202
   lookml_parser.py2431693%15–17, 86, 93–97, 112–113, 139, 180–181, 207–208, 272, 557
metaphor/manual/governance
   init.py4175%6
   extractor.py28293%24, 27
metaphor/manual/lineage
   init.py4175%6
   extractor.py31294%24, 27
metaphor/metabase
   init.py4175%6
   extractor.py1638945%47, 50, 70, 99–150, 153–162, 165–170, 178, 181, 188–189, 195–237, 253–256, 263, 268–296, 306, 312, 314, 318, 331–333
metaphor/postgresql
   init.py4175%6
   extractor.py1779944%9–11, 47, 50, 61–81, 87–88, 98–105, 113–157, 162–210, 213–259, 269–293, 297–302, 313–329, 374
metaphor/postgresql/profile
   init.py4175%6
   extractor.py1214662%7–9, 32, 35, 39, 44–47, 53–71, 79–95, 98–113, 205–206, 225–226, 233–235
metaphor/postgresql/usage
   init.py440%1–6
   config.py550%1–8
   extractor.py46460%1–78
metaphor/power_bi
   init.py4175%6
   extractor.py30611363%47–49, 169, 172–185, 189–190, 196–197, 203–204, 210–211, 217–218, 224–225, 230–292, 306–315, 322, 325, 342, 361–362, 403–404, 429–430, 441–442, 479, 492–493, 537–582
   power_query_parser.py113992%63, 127–132, 176–179
metaphor/redshift
   init.py4175%6
   access_event.py39685%53, 74–85
   extractor.py432737%18, 21, 25, 28–29, 32–52, 55–74
metaphor/redshift/lineage
   init.py4175%6
   extractor.py59395%28, 31, 35
metaphor/redshift/profile
   init.py440%1–6
   config.py550%1–8
   extractor.py23230%1–36
metaphor/redshift/query
   init.py440%1–6
   config.py990%1–18
   extractor.py42420%1–88
metaphor/redshift/usage
   init.py4175%6
   extractor.py47589%22, 25, 29, 67, 70
metaphor/snowflake
   auth.py381074%14–16, 61–80
   extractor.py18214321%18–20, 45, 48, 52, 55–57, 60–98, 102–105, 117–143, 146–175, 189–230, 235–263, 266–289, 292–313, 317, 327–355
   utils.py613248%42–56, 71–92, 103–125
metaphor/snowflake/lineage
   init.py4175%6
   extractor.py863460%38, 41, 45, 57–130, 133–138, 157, 167, 173, 177–178, 183–184
metaphor/snowflake/profile
   init.py4175%6
   extractor.py1576360%8–10, 49, 52, 56, 59–63, 68–97, 106–133, 148–189
metaphor/snowflake/query
   init.py440%1–6
   config.py11110%1–23
   extractor.py75750%1–193
metaphor/snowflake/usage
   init.py4175%6
   extractor.py843558%38, 41, 45, 58–128, 131–133, 146, 151–152, 156–157, 173, 184–185
metaphor/tableau
   init.py4175%6
   extractor.py1486159%16–18, 59, 62, 75, 78–144, 191–193, 218–221, 265, 271, 277, 281–285, 291, 313–327, 350
metaphor/thought_spot
   init.py440%1–6
   config.py14140%1–23
   extractor.py1181180%1–324
   models.py1191190%4–192
   utils.py95950%1–257
TOTAL9690351664% 

Tests Skipped Failures Errors Time
124 0 💤 0 ❌ 0 🔥 7.021s ⏱️

@mars-lan mars-lan marked this pull request as ready for review August 2, 2022 00:15
Copy link
Contributor

@elic-eon elic-eon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mars-lan mars-lan enabled auto-merge (squash) August 2, 2022 12:40
@mars-lan mars-lan merged commit b59c02c into main Aug 2, 2022
@mars-lan mars-lan deleted the marslan/sc-10762/replace-crawler-specific-entry-points-with branch August 2, 2022 12:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants