feat: Add Tableau dashboard metadata extractors #333

ccarterlandis · 2020-08-14T21:52:15Z

Summary of Changes

This PR introduces 5 extractors for Tableau dashboard metadata:

TableauDashboardMetadataExtractor
TableauDashboardTableExtractor
TableauDashboardQueryExtractor
TableauDashboardLastModifiedtExtractor
TableauExternalTableExtractor

See amundsen-io/amundsen#565 for the related feature proposal.

Tests

Basic unit tests were added for all extractors except the external table extractor. Tests for Tableau authencation and the utility methods have not been created yet.

Documentation

Added sections for each of the extractors to the README.md.

CheckList

Make sure you have checked all steps below to ensure a timely review.

PR title addresses the issue accurately and concisely. Example: "Updates the version of Flask to v1.0.2"
- In case you are adding a dependency, check if the license complies with the ASF 3rd Party License Policy.
PR includes a summary of changes.
PR adds unit tests, updates existing unit tests, OR documents why no test additions or modifications are needed.
In case of new functionality, my PR adds documentation that describes how to use it.
- All the public functions and the classes in the PR contain docstrings that explain what it does
[] PR passes make test

TODO

Fix any linting errors
Make sure this PR passes make test

ccarterlandis · 2020-08-14T21:55:29Z

@feng-tao I'm guessing those upstream commits + merge conflicts are because master has changed since I created this PR. This PR should remain draft until the linting errors have been fixed and & make test passes, after that it can be opened for review

ccarterlandis · 2020-08-14T21:56:28Z

Tagging for visibility 🚀 @alevene @GustoFergy

alevene · 2020-08-19T15:27:25Z

@dorianj I've applied the typing changes. Please review, thanks!

feng-tao · 2020-08-19T17:00:45Z

if the pr is ready for review, would be good to change the pr status.

alevene · 2020-08-19T17:06:13Z

@feng-tao could you do this for me? I'm getting this note:

Only those with write access to this repository can mark a draft pull request as ready for review.

feng-tao · 2020-08-19T17:07:34Z

@feng-tao could you do this for me? I'm getting this note:

Only those with write access to this repository can mark a draft pull request as ready for review.

maybe because the pr is created by carter? Anyway I have opened the pr from draft to review :)

dorianj · 2020-08-19T18:27:07Z

tests/unit/extractor/dashboard/tableau/test_tableau_dashboard_extractor.py

+import unittest
+
+from mock import patch
+from pyhocon import ConfigFactory  # noqa: F401


nit: don't add new noqas, these aren't needed anymore

Missed these, will fix. Thanks

dorianj · 2020-08-19T18:30:49Z

tests/unit/extractor/dashboard/tableau/test_tableau_dashboard_extractor.py

+logging.basicConfig(level=logging.INFO)
+
+
+def mock_query(*_args, **_kwargs):


I'm surprised that mypy isn't mad about there not being a return type on this, I have no idea why it's not... but it doesn't matter here

Oh, I know why. There was a mixup along the way and our branch did a rename of tests/unit/extractor/dashboard/__init__.py to tests/unit/extractor/dashboard/tableau/__init__.py. So, it's not treated as a module and mypy skips it.
I'll fix.

dorianj · 2020-08-19T18:34:18Z

@dorianj I've applied the typing changes. Please review, thanks!

Typings LGTM!

feng-tao · 2020-08-22T22:44:27Z

@alevene @ccarterlandis I will go over the pr early next week.

feng-tao · 2020-08-22T22:46:24Z

databuilder/models/dashboard/dashboard_table.py

@@ -57,7 +57,7 @@ def create_next_relation(self) -> Union[Dict[str, Any], None]:

    def _create_relation_iterator(self) -> Optional[Iterator[Dict[str, Any]]]:
        for table_id in self._table_ids:
-            m = re.match('(\w+)://(\w+)\.(\w+)\/(\w+)', table_id)
+            m = re.match(r'(\w+)://(\w+)\.(\w+)\/(\w+)', table_id)


could you rebase this line to r'([^./]+)://([^./]+)\.([^./]+)\/([^./]+)'

feng-tao

the code is very well written, lgtm. Let me know once you rebased. thanks @alevene @ccarterlandis

feng-tao · 2020-08-24T16:24:14Z

databuilder/extractor/dashboard/tableau/tableau_dashboard_query_extractor.py

+    def execute(self) -> Iterator[Dict[str, Any]]:
+        response = self.execute_query()
+
+        for query in response['customSQLTables']:


a comment on which API it is using?

Could you clarify what you'd like to see? There is a comment for this class that calls out what model is expected (DashboardQuery). The API relates to the query found in the TableauDashboardQueryExtractor class in this file.

I see, I miss that part. No worry then.

feng-tao · 2020-08-24T16:33:04Z

example/scripts/sample_tableau_data_loader.py

@@ -0,0 +1,366 @@
+"""


could you add the header here as well?

This patch adds a very barebones Tableau dashboard integration. Currently, it supports adding workbooks, users and owners (which are fake), and finding the last updated timestamp. To run it, startup Amundsen as normal and then run `python example/scripts/sample_data_loader.py`. You'll need to make sure that you've added your Tableau site config and credentials to the top of the sample loader script. It also does not support creation of tables and columns at this time, so you'll need to add those manually using a CSV job. Signed-off-by: Carter Landis <carter.landis@gusto.com>

This patch introduces a new Tableau dashboard extractor for gathering the queries used in Tableau workbooks. In Tableau, these are classified as "custom SQL tables", and appear as a datasource for the sheets in which they are used. Luckily, the Tableau Metadata API is smart enough to include the tables used in a custom SQL query in the workbook <> table relation. Signed-off-by: Carter Landis <carter.landis@gusto.com>

Signed-off-by: Carter Landis <carter.landis@gusto.com>

Signed-off-by: Carter Landis <carter.landis@gusto.com> (cherry picked from commit e8284e0935ea60eb65733433f1cba6c866466fbc) Signed-off-by: Carter Landis <carter.landis@gusto.com>

Signed-off-by: Carter Landis <carter.landis@gusto.com> (cherry picked from commit 80196e534b402db863a62e6d381e037a5ff95337) Signed-off-by: Carter Landis <carter.landis@gusto.com>

The external table extractor needs to use query variables in order to filter which external data connections are captured as external tables. (cherry picked from commit 32462fab306b117ca41c3e05593f7f124fe51efd) Signed-off-by: Carter Landis <carter.landis@gusto.com>

Signed-off-by: Carter Landis <carter.landis@gusto.com> (cherry picked from commit d587624aab55b5a950c4791d706a6bcc892c7ee0) Signed-off-by: Carter Landis <carter.landis@gusto.com>

Signed-off-by: Carter Landis <carter.landis@gusto.com> (cherry picked from commit 84b1ed91591d54ad34b7e54e843923634321e9b9) Signed-off-by: Carter Landis <carter.landis@gusto.com>

(cherry picked from commit 124ccb388f9d8d28ecc1b5a43eeb71a825e63ce0) Signed-off-by: Carter Landis <carter.landis@gusto.com>

Signed-off-by: Carter Landis <carter.landis@gusto.com> (cherry picked from commit ed8e0a6a9b2b1968f63027045464b47de5886100) Signed-off-by: Carter Landis <carter.landis@gusto.com>

…sanitize methods Signed-off-by: Carter Landis <carter.landis@gusto.com>

Signed-off-by: Carter Landis <carter.landis@gusto.com>

Python 3.8 is smart enough to understand named string interpolation with out the call to .format() added by this patch. Python 3.7 is not smart enough to make the same assumption. Signed-off-by: Carter Landis <carter.landis@gusto.com>

Signed-off-by: Carter Landis <carter.landis@gusto.com>

Signed-off-by: Alex Levene <alex.levene@gusto.com>

…ter module re-instated Signed-off-by: Alex Levene <alex.levene@gusto.com>

Signed-off-by: Alex Levene <alex.levene@gusto.com>

feng-tao

lgtm! thanks @alevene @ccarterlandis for the great contribution!
could you also add Tableau icon like amundsen-io/amundsenfrontendlibrary#501 into frontend repo
and update https://github.com/amundsen-io/amundsen#dashboard-connectors to include tableau? thanks a lot

ccarterlandis mentioned this pull request Aug 14, 2020

feat (WIP): Spike out PoC for Tableau integration #303

Closed

ccarterlandis force-pushed the tableau-dashboard-extractors branch from 760f388 to d7b9bb8 Compare August 14, 2020 21:53

alevene force-pushed the tableau-dashboard-extractors branch 2 times, most recently from b2bf2fb to 84326fb Compare August 18, 2020 21:58

feng-tao marked this pull request as ready for review August 19, 2020 17:06

dorianj reviewed Aug 19, 2020

View reviewed changes

feng-tao reviewed Aug 22, 2020

View reviewed changes

feng-tao reviewed Aug 24, 2020

View reviewed changes

ccarterlandis added 13 commits August 24, 2020 12:14

Fix linting errors in Tableau dashboard extractor and utils module

bf3d6d8

Signed-off-by: Carter Landis <carter.landis@gusto.com>

Fix linting errors for Tableau last modified extractor

04e2c7b

Signed-off-by: Carter Landis <carter.landis@gusto.com>

Fix linting errors in Tableau owner extractor

258f16c

Signed-off-by: Carter Landis <carter.landis@gusto.com>

Fix linting errors in Tableau user extractor

f495b72

Signed-off-by: Carter Landis <carter.landis@gusto.com>

Fix linting errors in Tableau query extractor

f0bb702

Signed-off-by: Carter Landis <carter.landis@gusto.com>

Fix linting errors in Tableau table extractor

2f018e9

Signed-off-by: Carter Landis <carter.landis@gusto.com>

Remove Tableau paginated REST API class

a0f5794

Signed-off-by: Carter Landis <carter.landis@gusto.com>

Fix linting errors in Tableau sample data loader

df6b4bc

Signed-off-by: Carter Landis <carter.landis@gusto.com>

Standardize Tableau access token naming

18f4233

Signed-off-by: Carter Landis <carter.landis@gusto.com>

Remove dummy Tableau user and owner extractors

82f3d5f

Signed-off-by: Carter Landis <carter.landis@gusto.com>

Update Tableau extractors scopes

3472259

Signed-off-by: Carter Landis <carter.landis@gusto.com>

ccarterlandis and others added 26 commits August 24, 2020 12:14

Add unit test for Tableau dashboard last modified extractor

234ff8c

Signed-off-by: Carter Landis <carter.landis@gusto.com> (cherry picked from commit e8284e0935ea60eb65733433f1cba6c866466fbc) Signed-off-by: Carter Landis <carter.landis@gusto.com>

Generate Tableau dashboard source URLs on extraction

f442cb1

Signed-off-by: Carter Landis <carter.landis@gusto.com> (cherry picked from commit 80196e534b402db863a62e6d381e037a5ff95337) Signed-off-by: Carter Landis <carter.landis@gusto.com>

Standardize Tableau extractor sanitization methods

59ac88a

Signed-off-by: Carter Landis <carter.landis@gusto.com> (cherry picked from commit d587624aab55b5a950c4791d706a6bcc892c7ee0) Signed-off-by: Carter Landis <carter.landis@gusto.com>

Update README.md with Tableau extractor examples

0c5d179

Signed-off-by: Carter Landis <carter.landis@gusto.com> (cherry picked from commit 84b1ed91591d54ad34b7e54e843923634321e9b9) Signed-off-by: Carter Landis <carter.landis@gusto.com>

Update Tableau extractors to use class constants for config keys

4d3e8eb

(cherry picked from commit 124ccb388f9d8d28ecc1b5a43eeb71a825e63ce0) Signed-off-by: Carter Landis <carter.landis@gusto.com>

Add unit test for Tableau dashboard table extractor

95ef36f

Signed-off-by: Carter Landis <carter.landis@gusto.com> (cherry picked from commit ed8e0a6a9b2b1968f63027045464b47de5886100) Signed-off-by: Carter Landis <carter.landis@gusto.com>

Enforce consistent use of dict literals, class config constants, and …

41b7ebe

…sanitize methods Signed-off-by: Carter Landis <carter.landis@gusto.com>

Move misplaced __init__.py file in Tableau tests

9af6449

Signed-off-by: Carter Landis <carter.landis@gusto.com>

Revise small Tableau documentation errors

34f349e

Signed-off-by: Carter Landis <carter.landis@gusto.com>

Fix minor formatting/linting errors in Tableau extractors

6886dbe

Signed-off-by: Carter Landis <carter.landis@gusto.com>

Remove unused config constants from Tableau extractors

0eb3bb4

Signed-off-by: Carter Landis <carter.landis@gusto.com>

Enforce superclass init signature for Tableau sub-extractors

da0eefe

Signed-off-by: Carter Landis <carter.landis@gusto.com>

Refactor Tableau GraphQL extractor to use fallback for query vars

43327dc

Signed-off-by: Carter Landis <carter.landis@gusto.com>

Move Tableau extractor auth to GraphQL extractor init

1a5ec02

Signed-off-by: Carter Landis <carter.landis@gusto.com>

Expose Tableau request verification as config

6c5343e

Signed-off-by: Carter Landis <carter.landis@gusto.com>

Add unit test for Tableau dashboard extractor

62df04f

Signed-off-by: Carter Landis <carter.landis@gusto.com>

Some more linting

4b33319

Signed-off-by: Alex Levene <alex.levene@gusto.com>

set config key interface for each extractor

518a129

Signed-off-by: Alex Levene <alex.levene@gusto.com>

Clean up and simplify sample loader

1e70cc5

Signed-off-by: Alex Levene <alex.levene@gusto.com>

Linting on tests

1c43ec3

Signed-off-by: Alex Levene <alex.levene@gusto.com>

Convert typings to mypy

b4da526

Signed-off-by: Alex Levene <alex.levene@gusto.com>

Dashboard test was not a module. Fix remaining typing issues found af…

a41ffd1

…ter module re-instated Signed-off-by: Alex Levene <alex.levene@gusto.com>

Documentation typos and import re-ordering

196a719

Signed-off-by: Alex Levene <alex.levene@gusto.com>

Add header to sample tableau data loader

49c9c8b

Signed-off-by: Alex Levene <alex.levene@gusto.com>

alevene force-pushed the tableau-dashboard-extractors branch from c19efe8 to 49c9c8b Compare August 24, 2020 19:08

feng-tao approved these changes Aug 24, 2020

View reviewed changes

feng-tao merged commit 46207ee into amundsen-io:master Aug 24, 2020

feng-tao mentioned this pull request Aug 26, 2020

feat: add Tableau icon to fe amundsen-io/amundsenfrontendlibrary#616

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add Tableau dashboard metadata extractors #333

feat: Add Tableau dashboard metadata extractors #333

ccarterlandis commented Aug 14, 2020

ccarterlandis commented Aug 14, 2020

ccarterlandis commented Aug 14, 2020

alevene commented Aug 19, 2020

feng-tao commented Aug 19, 2020

alevene commented Aug 19, 2020 •

edited

feng-tao commented Aug 19, 2020

dorianj Aug 19, 2020

alevene Aug 19, 2020

dorianj Aug 19, 2020

alevene Aug 19, 2020

dorianj commented Aug 19, 2020

feng-tao commented Aug 22, 2020

feng-tao Aug 22, 2020

feng-tao left a comment

feng-tao Aug 24, 2020

alevene Aug 24, 2020

feng-tao Aug 24, 2020

feng-tao Aug 24, 2020

feng-tao left a comment

		logging.basicConfig(level=logging.INFO)


		def mock_query(_args, *_kwargs):

feat: Add Tableau dashboard metadata extractors #333

feat: Add Tableau dashboard metadata extractors #333

Conversation

ccarterlandis commented Aug 14, 2020

Summary of Changes

Tests

Documentation

CheckList

TODO

ccarterlandis commented Aug 14, 2020

ccarterlandis commented Aug 14, 2020

alevene commented Aug 19, 2020

feng-tao commented Aug 19, 2020

alevene commented Aug 19, 2020 • edited

feng-tao commented Aug 19, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dorianj commented Aug 19, 2020

feng-tao commented Aug 22, 2020

Choose a reason for hiding this comment

feng-tao left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

feng-tao left a comment

Choose a reason for hiding this comment

alevene commented Aug 19, 2020 •

edited