Skip to content
This repository has been archived by the owner on Nov 30, 2022. It is now read-only.

706 Adds SaaS connection type to SaaS yaml config #748

Merged
merged 25 commits into from
Jul 6, 2022

Conversation

eastandwestwind
Copy link
Contributor

@eastandwestwind eastandwestwind commented Jun 28, 2022

Purpose

Adds SaaS connection type to SaaS yaml config

Changes

  • Adds SaaS connection type to SaaS yaml config
  • Updates test for invalid saas type
  • Updates docs and postman collection

Checklist

  • Update CHANGELOG.md file
    • Merge in main so the most recent CHANGELOG.md file is being appended to
    • Add description within the Unreleased section in an appropriate category. Add a new category from the list at the top of the file if the needed one isn't already there.
    • Add a link to this PR at the end of the description with the PR number as the text. example: #1
  • Applicable documentation updated (guides, quickstart, postman collections, tutorial, fidesdemo, database diagram.
  • If docs updated (select one):
    • documentation complete, or draft/outline provided (tag docs-team to complete/review on this branch)
    • documentation issue created (tag docs-team to complete issue separately)
  • Good unit test/integration test coverage
  • This PR contains a DB migration. If checked, the reviewer should confirm with the author that the down_revision correctly references the previous migration before merging
  • The Run Unsafe PR Checks label has been applied, and checks have passed, if this PR touches any external services

Ticket

Fixes #706

@pattisdr pattisdr self-assigned this Jun 30, 2022
Copy link
Contributor

@pattisdr pattisdr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice job in trying to find a workable solution for everyone @eastandwestwind, just a few minor things here.

fidesops.toml Outdated Show resolved Hide resolved
src/fidesops/schemas/saas/saas_config.py Outdated Show resolved Hide resolved
src/fidesops/schemas/saas/saas_config.py Outdated Show resolved Hide resolved
src/fidesops/schemas/saas/saas_config.py Show resolved Hide resolved
@eastandwestwind
Copy link
Contributor Author

eastandwestwind commented Jul 6, 2022

@pattisdr I've been struggling with a CI error during init-db related to the new migration. Seems that it may be expecting an enum for ConnectionType with raw SQL:

sqlalchemy.exc.OperationalError: (psycopg2.errors.UnsafeNewEnumValueUsage) unsafe use of new value "saas" of enum type connectiontype
LINE 1: ...on_config_id WHERE connectionconfig.connection_type = 'saas'
                                                                 ^
HINT:  New enum values must be committed before they can be used.

E.g. https://github.com/ethyca/fidesops/runs/7206676793?check_suite_focus=true

I've been testing out some theories, but let me know if you've seen this before

eastandwestwind and others added 5 commits July 6, 2022 08:15
…, update previous schema migrations that used "alter type" to update connection type to rename the enum type, create a new enum with the new types, and then delete the old enum type.

- Update the data migration to update the connection config > saas config field instead of the dataset config field.
@pattisdr
Copy link
Contributor

pattisdr commented Jul 6, 2022

Unit tests were failing because we were using alter type to incrementally add to the connection_type enum but in this newest migration, we wanted to use that new enum, but it hadn't yet been commited.

I've updated older migrations to not use alter type, when updating the connectino_type enum:
https://www.postgresql.org/docs/14/sql-altertype.html

If ALTER TYPE ... ADD VALUE (the form that adds a new value to an enum type) is executed inside a transaction block, the new value cannot be used until after the transaction has been committed.

CHANGELOG.md Outdated Show resolved Hide resolved
src/fidesops/schemas/saas/saas_config.py Outdated Show resolved Hide resolved
@eastandwestwind
Copy link
Contributor Author

@pattisdr I've just pushed out the changelog update (looks like some other recent PRs have been added to the 1.6.1 section also 😢 ). Otherwise, your changes look good, thanks so much for helping me out with this!

@@ -19,6 +19,10 @@ The types of changes are:

## [Unreleased](https://github.com/ethyca/fidesops/compare/1.6.1...main)

### Added
* Adds SaaS connection type to SaaS yaml config [748](https://github.com/ethyca/fidesops/pull/748)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you sure this is updated with main? the changelog there has more items in unreleased?

https://github.com/ethyca/fidesops/blob/c5c30840d07d2c2c4b999bdf241044690c63d092/CHANGELOG.md

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

weirdddd lemme try again

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, this looks better now

@pattisdr pattisdr merged commit 511835b into main Jul 6, 2022
@pattisdr pattisdr deleted the 706-datastore-type-saas branch July 6, 2022 17:12
sanders41 pushed a commit that referenced this pull request Sep 22, 2022
* Adds Saas type to saas yaml config

* To allow data migration that queries on a connectiontype enum to work, update previous schema migrations that used "alter type" to update connection type to rename the enum type, create a new enum with the new types, and then delete the old enum type.

Co-authored-by: Dawn Pattison <pattisdr@users.noreply.github.com>
@pattisdr pattisdr mentioned this pull request Sep 22, 2022
10 tasks
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Datastore Management] Include Icon with datastore connection (BE)
3 participants