feat: Add Private Google Sheets to dynamic form #16628

AAfghahi · 2021-09-08T00:12:38Z

SUMMARY

This PR does two things, firstly it adds support for private gsheets as a dynamic form in the DatabaseConnection modal. Secondly, it refactors how we pass encrypted fields in said modal.

Previously this sections were very narrow. What this PR does is, during the onSave action after all parameters have been validated, goes through the dbModel or a new calculated column named parameters_schema and checks to see if each parameter is encrypted. It uses the 'x-encrypted-extra' field that is exposed through the /available endpoint.

One point of confusion that might arise from reading the code, we pass in encrypted fields as strings in the frontend, but then convert them to dictionaries in the backend, thus when fetching encrypted fields, the fields have to be serialized in order to be properly saved.

Currently the only other database that uses encrypted fields is Big Query, the previous logic used on creation and edit have been preserved and are applied to gsheets.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

adding a private sheet:

Screen.Recording.2021-09-23.at.5.58.47.PM.mov

adding a private sheet to a existing database that has only public sheets:

Screen.Recording.2021-09-23.at.5.55.25.PM.mov

TESTING INSTRUCTIONS

I am able to create any database that is able to be added dynamically. I tested this on Gsheets, BigQuery, and Postgres (using a very basic localhost postgres).

I was able to add a private gsheet. Am able to go back and edit it, and still be able to connect and finish.

Create a database that only has public gsheets.

Create a database that only has public gsheets, and then when editing it add a private gsheet with a proper encrypted json.

ADDITIONAL INFORMATION

Has associated issue:
Required feature flags:
Changes UI
Includes DB Migration (follow approval process in SIP-59)
- Migration is atomic, supports rollback & is backwards-compatible
- Confirm DB migration upgrade and downgrade tested
- Runtime estimates and downtime expectations provided
Introduces new feature or API
Removes existing feature or API

codecov · 2021-09-23T16:09:55Z

Codecov Report

Merging #16628 (0ddfc96) into master (b35645c) will decrease coverage by 0.21%.
The diff coverage is 38.35%.

❗ Current head 0ddfc96 differs from pull request most recent head f594706. Consider uploading reports for the commit f594706 to get more accurate results

@@            Coverage Diff             @@
##           master   #16628      +/-   ##
==========================================
- Coverage   77.05%   76.83%   -0.22%     
==========================================
  Files        1021     1021              
  Lines       54693    54709      +16     
  Branches     7457     7457              
==========================================
- Hits        42141    42033     -108     
- Misses      12307    12431     +124     
  Partials      245      245

Flag	Coverage Δ
hive	`?`
mysql	`81.84% <78.26%> (-0.01%)`	⬇️
postgres	`81.86% <78.26%> (-0.05%)`	⬇️
presto	`?`
python	`81.99% <78.26%> (-0.43%)`	⬇️
sqlite	`81.48% <78.26%> (-0.05%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
superset/databases/api.py	`92.95% <ø> (ø)`
...c/views/CRUD/data/database/DatabaseModal/index.tsx	`44.13% <13.04%> (ø)`
.../database/DatabaseModal/DatabaseConnectionForm.tsx	`53.92% <20.83%> (ø)`
superset/db_engine_specs/bigquery.py	`84.71% <37.50%> (-2.13%)`	⬇️
...c/views/CRUD/data/database/DatabaseModal/styles.ts	`81.57% <50.00%> (ø)`
...set-frontend/src/views/CRUD/data/database/types.ts	`100.00% <100.00%> (ø)`
superset/databases/schemas.py	`98.40% <100.00%> (+0.01%)`	⬆️
superset/db_engine_specs/gsheets.py	`75.00% <100.00%> (+0.26%)`	⬆️
superset/models/core.py	`89.26% <100.00%> (-0.56%)`	⬇️
superset/db_engines/hive.py	`0.00% <0.00%> (-82.15%)`	⬇️
... and 11 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b35645c...f594706. Read the comment docs.

betodealmeida

Looks good, I just have a few questions. Also, let's leave all those empty lines you deleted, they help with readability.

superset-frontend/src/views/CRUD/data/database/DatabaseModal/DatabaseConnectionForm.tsx

superset/models/core.py

betodealmeida · 2021-09-23T19:40:10Z

superset-frontend/src/views/CRUD/data/database/DatabaseModal/DatabaseConnectionForm.tsx

+  const [isPublic, setIsPublic] = useState<boolean>(true);
+  const showCredentialsInfo =
+    db?.engine === 'gsheets' ? !isEditMode && !isPublic : !isEditMode;
+  const isEncrypted = db?.encrypted_extra && db?.encrypted_extra?.length > 2;


Why isn't an encrypted_extra field with a single key/value pair considered encrypted?

in the backend we make encrypted_extra into a string of an empty object. So "{}".

betodealmeida · 2021-09-23T19:47:57Z

superset-frontend/src/views/CRUD/data/database/DatabaseModal/DatabaseConnectionForm.tsx

 enum CredentialInfoOptions {
  jsonUpload,
  copyPaste,
 }

+const setStringToBoolean = (optionValue: string) => {


Suggested change

const setStringToBoolean = (optionValue: string) => {

const castStringToBoolean = (optionValue: string) => {

betodealmeida · 2021-09-23T19:53:27Z

superset-frontend/src/views/CRUD/hooks.ts

@@ -277,7 +277,6 @@ export function useSingleViewResource<D extends object = any>(
      updateState({
        loading: true,
      });
-


Code also needs to breath... let's leave these empty lines! Usually empty lines separate logical blocks that are (slightly) unrelated.

betodealmeida · 2021-09-23T19:53:39Z

superset/databases/api.py

@@ -920,7 +921,6 @@ def available(self) -> Response:
        for engine_spec, drivers in get_available_engine_specs().items():
            if not drivers:
                continue
-


betodealmeida · 2021-09-23T19:53:44Z

superset/databases/api.py

@@ -943,7 +943,6 @@ def available(self) -> Response:
                payload[
                    "sqlalchemy_uri_placeholder"
                ] = engine_spec.sqlalchemy_uri_placeholder  # type: ignore
-


Ditto. Is your editor automatically removing these lines?

superset/db_engine_specs/gsheets.py

hughhhh · 2021-09-24T16:47:23Z

/testenv up

github-actions · 2021-09-24T16:49:29Z

@hughhhh Ephemeral environment spinning up at http://34.222.153.226:8080. Credentials are admin/admin. Please allow several minutes for bootstrapping and startup.

betodealmeida

This looks great, I left just a few comments.

betodealmeida · 2021-09-24T17:40:31Z

superset-frontend/src/views/CRUD/data/database/DatabaseModal/DatabaseConnectionForm.tsx

+            css={(theme: SupersetTheme) => labelMarginBotton(theme)}
+            required
+          >
+            {t('Type of Google Sheets Allowed')}


Small nit, let's keep the copy consistent in terms of capitalization:

Suggested change

{t('Type of Google Sheets Allowed')}

{t('Type of Google Sheets allowed')}

betodealmeida · 2021-09-24T18:00:50Z

superset-frontend/src/views/CRUD/data/database/DatabaseModal/DatabaseConnectionForm.tsx

+              <FormLabel required>{t('Upload Credentials')}</FormLabel>
+              <InfoTooltip
+                tooltip={t(
+                  'Use the JSON file you automatically downloaded when creating your service account in Google BigQuery.',


This now applies to GSheets as well, right?

Suggested change

'Use the JSON file you automatically downloaded when creating your service account in Google BigQuery.',

'Use the JSON file you automatically downloaded when creating your service account.',

Thanks for catching this, originally I had a separate component for Gsheets, but I forgot to go and update this when I combined all the encrypted fields.

betodealmeida · 2021-09-24T18:05:04Z

superset-frontend/src/views/CRUD/data/database/DatabaseModal/index.tsx

-      if (engine === 'bigquery' && dbToUpdate.parameters?.credentials_info) {
-        // wrap encrypted_extra in credentials_info only for BigQuery
+      paramConfigArray.forEach(paramConfig => {
+        // we are going through the parameter configuration and seeing if this is an encrypted field


A suggestion for comments: good comments describe why something is being done, instead of what is being done. Something like this:

/* * Parameters that are annotated with the `x-encrypted-extra` properties should be moved to * `encrypted_extra`, so that they are stored encrypted in the backend when the database is * created or edited. */

betodealmeida · 2021-09-24T18:08:23Z

superset-frontend/src/views/CRUD/data/database/DatabaseModal/index.tsx

+            // add new encrypted extra to encrypted_extra object
+            additionalEncryptedExtra[paramConfig] =
+              dbToUpdate.parameters?.[paramConfig];
+            // cast the encrypted field as a string in parameters


Similarly here and in line 549: it's clear that we're casting it to a string, so the comment doesn't add much. It would be better to say something like:

// The backend expects `encrypted_extra` as a string for historical reasons.

betodealmeida · 2021-09-24T18:11:47Z

superset/models/core.py

+    @property
+    def parameters_schema(self) -> Dict[str, Any]:
+        try:
+            parameters_schema = self.db_engine_spec.parameters_json_schema()  # type: ignore # pylint: disable=line-too-long,useless-suppression


This line can be cleaned up a little bit. If you're disabling the useless-suppression lint rule it means there's another rule you're suppressing unnecessarily. You can probably write this as (untested):

Suggested change

parameters_schema = self.db_engine_spec.parameters_json_schema() # type: ignore # pylint: disable=line-too-long,useless-suppression

parameters_schema = self.db_engine_spec.parameters_json_schema() # type: ignore

when I had this with just type: ignore it was failing the python lint test. So I emulated this line:

https://github.com/apache/superset/blob/master/superset/models/core.py#L246

betodealmeida

This looks great! Thanks for all the work, @AAfghahi !

github-actions · 2021-09-29T00:09:15Z

Ephemeral environment shutdown and build artifacts deleted.

* first pass private gsheets * made encrypted extra into string, refactored onParametersChanged * private sheets working, credential_info errors * all but test connection working * first pass private gsheets * made encrypted extra into string, refactored onParametersChanged * private sheets working, credential_info errors * all but test connection working * Regenerate package-lock.json Co-authored-by: Beto Dealmeida <roberto@dealmeida.net>

eschutho · 2021-10-06T17:53:40Z

@AAfghahi I don't see a package.json file change in here.. was the package-lock an oversight?

AAfghahi · 2021-10-06T17:58:02Z

No, it was failing e2e tests so Beto regenerated the package-lock here: e56ea3f

so that it would pass the tests.

* first pass private gsheets * made encrypted extra into string, refactored onParametersChanged * private sheets working, credential_info errors * all but test connection working * first pass private gsheets * made encrypted extra into string, refactored onParametersChanged * private sheets working, credential_info errors * all but test connection working * Regenerate package-lock.json Co-authored-by: Beto Dealmeida <roberto@dealmeida.net>

pull-request-size bot added the size/M label Sep 8, 2021

AAfghahi force-pushed the ch18781_privateGSheets branch 2 times, most recently from 6c0dea2 to aee5355 Compare September 8, 2021 17:01

pull-request-size bot added size/L and removed size/M labels Sep 8, 2021

AAfghahi force-pushed the ch18781_privateGSheets branch 2 times, most recently from c201af7 to f7cc7ac Compare September 13, 2021 20:18

AAfghahi force-pushed the ch18781_privateGSheets branch 5 times, most recently from 2221c0c to 40662ea Compare September 23, 2021 15:41

AAfghahi marked this pull request as ready for review September 23, 2021 15:45

AAfghahi force-pushed the ch18781_privateGSheets branch 3 times, most recently from af2c6be to 34ad067 Compare September 23, 2021 19:12

betodealmeida requested changes Sep 23, 2021

View reviewed changes

AAfghahi force-pushed the ch18781_privateGSheets branch 8 times, most recently from f49bb8c to 07baa49 Compare September 23, 2021 22:00

betodealmeida requested changes Sep 24, 2021

View reviewed changes

AAfghahi force-pushed the ch18781_privateGSheets branch from 07baa49 to 6bfa79e Compare September 24, 2021 18:37

AAfghahi force-pushed the ch18781_privateGSheets branch 2 times, most recently from dca4017 to 4b42aba Compare September 24, 2021 20:05

betodealmeida approved these changes Sep 28, 2021

View reviewed changes

AAfghahi force-pushed the ch18781_privateGSheets branch from 4b42aba to 2eb3834 Compare September 28, 2021 23:11

betodealmeida force-pushed the ch18781_privateGSheets branch from 2eb3834 to e56ea3f Compare September 28, 2021 23:14

AAfghahi and others added 9 commits September 28, 2021 19:22

first pass private gsheets

08886ab

made encrypted extra into string, refactored onParametersChanged

ae0c4de

private sheets working, credential_info errors

11de3bc

all but test connection working

a29b75a

first pass private gsheets

fdffbba

made encrypted extra into string, refactored onParametersChanged

6c41a59

private sheets working, credential_info errors

7eae7c0

all but test connection working

00c0a73

Regenerate package-lock.json

f594706

AAfghahi force-pushed the ch18781_privateGSheets branch from e56ea3f to f594706 Compare September 28, 2021 23:35

betodealmeida merged commit aa74721 into apache:master Sep 29, 2021

siegfriedweber mentioned this pull request Feb 9, 2022

Add Superset 1.4.1 stackabletech/docker-images#47

Closed

mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 1.4.0 labels Mar 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add Private Google Sheets to dynamic form #16628

feat: Add Private Google Sheets to dynamic form #16628

AAfghahi commented Sep 8, 2021 •

edited

Loading

codecov bot commented Sep 23, 2021 •

edited

Loading

betodealmeida left a comment

betodealmeida Sep 23, 2021

AAfghahi Sep 23, 2021

betodealmeida Sep 23, 2021

betodealmeida Sep 23, 2021

betodealmeida Sep 23, 2021

betodealmeida Sep 23, 2021

hughhhh commented Sep 24, 2021

github-actions bot commented Sep 24, 2021

betodealmeida left a comment

betodealmeida Sep 24, 2021

betodealmeida Sep 24, 2021

AAfghahi Sep 24, 2021

betodealmeida Sep 24, 2021

betodealmeida Sep 24, 2021

betodealmeida Sep 24, 2021

AAfghahi Sep 24, 2021

betodealmeida left a comment

github-actions bot commented Sep 29, 2021

eschutho commented Oct 6, 2021

AAfghahi commented Oct 6, 2021

	const setStringToBoolean = (optionValue: string) => {
	const castStringToBoolean = (optionValue: string) => {

	{t('Type of Google Sheets Allowed')}
	{t('Type of Google Sheets allowed')}

	'Use the JSON file you automatically downloaded when creating your service account in Google BigQuery.',
	'Use the JSON file you automatically downloaded when creating your service account.',

	parameters_schema = self.db_engine_spec.parameters_json_schema() # type: ignore # pylint: disable=line-too-long,useless-suppression
	parameters_schema = self.db_engine_spec.parameters_json_schema() # type: ignore

feat: Add Private Google Sheets to dynamic form #16628

feat: Add Private Google Sheets to dynamic form #16628

Conversation

AAfghahi commented Sep 8, 2021 • edited Loading

SUMMARY

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

codecov bot commented Sep 23, 2021 • edited Loading

Codecov Report

betodealmeida left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hughhhh commented Sep 24, 2021

github-actions bot commented Sep 24, 2021

betodealmeida left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

betodealmeida left a comment

Choose a reason for hiding this comment

github-actions bot commented Sep 29, 2021

eschutho commented Oct 6, 2021

AAfghahi commented Oct 6, 2021

AAfghahi commented Sep 8, 2021 •

edited

Loading

codecov bot commented Sep 23, 2021 •

edited

Loading