Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Migrate encrypted data in database to new encryption structure #3315

Open
dmjb opened this issue May 13, 2024 · 0 comments · Fixed by #3356 · May be fixed by #3435
Open

Migrate encrypted data in database to new encryption structure #3315

dmjb opened this issue May 13, 2024 · 0 comments · Fixed by #3356 · May be fixed by #3435
Assignees
@dmjb
Labels
user-story

Comments

@dmjb
Copy link
Member

dmjb commented May 13, 2024

Please describe the enhancement

See design doc in epic

Solution Proposal

In migration tool:

  1. Change decrypt code to read from new fields then old fields
  2. Begin writing new secrets to new structure only
  3. Get each secret
  4. Decrypt
  5. Re-salt and re-encrypt using new structure
  6. Validate that migration has worked and drop old column

Describe alternatives you've considered

No response

Additional context

No response

Acceptance Criteria

No response
dmjb added a commit that referenced this issue May 14, 2024
@dmjb
Add new column for new encyrption format
b422a5f 
Relates to #3315

Add new JSONB columns to session store and access tokens table which
will be used to store encrypted data along with its metadata (algorithm
type, key, salt).
dmjb added a commit that referenced this issue May 14, 2024
@dmjb
Add new column for new encyrption format
a9d2ac0 
Relates to #3315

Add new JSONB columns to session store and access tokens table which
will be used to store encrypted data along with its metadata (algorithm
type, key, salt).
@dmjb dmjb mentioned this issue May 14, 2024
Merged
10 tasks
@dmjb dmjb self-assigned this May 16, 2024
dmjb added a commit that referenced this issue May 16, 2024
@dmjb
Store encrypted values in new DB column.
75d4df0 
Relates to #3315

When encrypting secrets, store the serialized JSON in the database. When
decrypting, attempt to fetch from the new column and then fall back to
the old one.

In a subsequent PR, I will migrate all existing secrets to the new
structure.
@dmjb dmjb mentioned this issue May 16, 2024
Merged
10 tasks
dmjb added a commit that referenced this issue May 16, 2024
@dmjb
Store encrypted values in new DB column.
3dacf36 
Relates to #3315

When encrypting secrets, store the serialized JSON in the database. When
decrypting, attempt to fetch from the new column and then fall back to
the old one.

In a subsequent PR, I will migrate all existing secrets to the new
structure.
dmjb added a commit that referenced this issue May 16, 2024
@dmjb
Store encrypted values in new DB column.
60f0387 
Relates to #3315

When encrypting secrets, store the serialized JSON in the database. When
decrypting, attempt to fetch from the new column and then fall back to
the old one.

In a subsequent PR, I will migrate all existing secrets to the new
structure.
dmjb added a commit that referenced this issue May 16, 2024
@dmjb
Store encrypted values in new DB column.
58c1982 
Relates to #3315

When encrypting secrets, store the serialized JSON in the database. When
decrypting, attempt to fetch from the new column and then fall back to
the old one.

In a subsequent PR, I will migrate all existing secrets to the new
structure.
dmjb added a commit that referenced this issue May 17, 2024
@dmjb
Store encrypted values in new DB column. (#3351)
c0286cd 
Relates to #3315

When encrypting secrets, store the serialized JSON in the database. When
decrypting, attempt to fetch from the new column and then fall back to
the old one.

In a subsequent PR, I will migrate all existing secrets to the new
structure.
@dmjb dmjb mentioned this issue May 17, 2024
Merged
10 tasks
dmjb added a commit that referenced this issue May 20, 2024
@dmjb
Stop storing secrets in old DB columns
aa5f08b 
Fixes #3315

At this point, we now have a hard requirement to use the new EncryptedData structure in the database - the old columns (for the access token, and for the redirect URL) do not track the random salt, and so they are useless for any new data. As part of the migration away from the old columns, I have made the following changes in this PR:

1. Make the old columns nullable
2. Stop writing to the old columns
3. Change all unit tests which relied on the old columns
dmjb added a commit that referenced this issue May 20, 2024
@dmjb
Add new TF variables for new crypto config
f376d97 
Relates to #3315

These new variables will be used to add the new crypto config for the
staging and production configs.
@dmjb dmjb mentioned this issue May 20, 2024
Merged
10 tasks
dmjb added a commit that referenced this issue May 21, 2024
@dmjb
Add new helm variables for new crypto config (#3378)
949b313 
Relates to #3315

These new variables will be used to add the new crypto config for the
staging and production configs.
@dmjb dmjb reopened this May 22, 2024
dmjb added a commit that referenced this issue May 22, 2024
@dmjb
Simplify crypto config structure
f870d16 
Relates to #3315

My first attempt at a config structure was a bad fit for the envionment
variable substitution which we used when deploying minder in k8s.
Simplify the crypto config in the following ways:

1) Only one fallback key/algo can be specified instead of a list. This
   is based under the assumption that we only need a single fallback
   before migrating to a new key.
2) The keystore config has been changed from a `map[string]any` to a
   concrete type. If we need more types of config in future, we can add
   additional concrete types and when filling in the config, only one
   type needs to be filled in.
@dmjb dmjb mentioned this issue May 22, 2024
Merged
10 tasks
dmjb added a commit that referenced this issue May 22, 2024
@dmjb
Simplify crypto config structure
2b39da9 
Relates to #3315

My first attempt at a config structure was a bad fit for the envionment
variable substitution which we used when deploying minder in k8s.
Simplify the crypto config in the following ways:

1) Only one fallback key/algo can be specified instead of a list. This
   is based under the assumption that we only need a single fallback
   before migrating to a new key.
2) The keystore config has been changed from a `map[string]any` to a
   concrete type. If we need more types of config in future, we can add
   additional concrete types and when filling in the config, only one
   type needs to be filled in.
dmjb added a commit that referenced this issue May 22, 2024
@dmjb
Simplify crypto config structure
3048e59 
Relates to #3315

My first attempt at a config structure was a bad fit for the envionment
variable substitution which we used when deploying minder in k8s.
Simplify the crypto config in the following ways:

1) Only one fallback key/algo can be specified instead of a list. This
   is based under the assumption that we only need a single fallback
   before migrating to a new key.
2) The keystore config has been changed from a `map[string]any` to a
   concrete type. If we need more types of config in future, we can add
   additional concrete types and when filling in the config, only one
   type needs to be filled in.
dmjb added a commit that referenced this issue May 22, 2024
@dmjb
Simplify crypto config structure
feb1eea 
Relates to #3315

My first attempt at a config structure was a bad fit for the envionment
variable substitution which we used when deploying minder in k8s.
Simplify the crypto config in the following ways:

1) Only one fallback key/algo can be specified instead of a list. This
   is based under the assumption that we only need a single fallback
   before migrating to a new key.
2) The keystore config has been changed from a `map[string]any` to a
   concrete type. If we need more types of config in future, we can add
   additional concrete types and when filling in the config, only one
   type needs to be filled in.
dmjb added a commit that referenced this issue May 22, 2024
@dmjb
Simplify crypto config structure
25fe404 
Relates to #3315

My first attempt at a config structure was a bad fit for the envionment
variable substitution which we used when deploying minder in k8s.
Simplify the crypto config in the following ways:

1) Only one fallback key/algo can be specified instead of a list. This
   is based under the assumption that we only need a single fallback
   before migrating to a new key.
2) The keystore config has been changed from a `map[string]any` to a
   concrete type. If we need more types of config in future, we can add
   additional concrete types and when filling in the config, only one
   type needs to be filled in.
dmjb added a commit that referenced this issue May 22, 2024
@dmjb
Simplify crypto config structure (#3404)
cb61ff9 
Relates to #3315

My first attempt at a config structure was a bad fit for the environment
variable substitution which we use when deploying minder in k8s.
Simplify the crypto config in the following ways:

1) Only one fallback key/algo can be specified instead of a list. This
   is based under the assumption that we only need a single fallback
   before migrating to a new key.
2) The keystore config has been changed from a `map[string]any` to a
   concrete type. If we need more types of config in future, we can add
   additional concrete types and when filling in the config, only one
   type needs to be filled in.
dmjb added a commit that referenced this issue May 27, 2024
@dmjb
Implement process for migrating keys and algorithms
1657961 
Fixes #3315

Implement CLI command which is used to migrate keys and algorithms. This
will:

1) Open connection to the DB
2) Query for secrets which do not have the default key ID or algorithm
3) Decrypts and re-encrypts with new data
4) Wraps all the above in a transaction which asks permission before
   commiting

State secrets in the session state table are deleted. Since they are
short-lived secrets, there is no point in migrating them.

Tested locally.
@dmjb dmjb linked a pull request May 27, 2024 that will close this issue
Open
10 tasks
dmjb added a commit that referenced this issue May 27, 2024
@dmjb
Implement process for migrating keys and algorithms
ca6e2e8 
Fixes #3315

Implement CLI command which is used to migrate keys and algorithms. This
will:

1) Open connection to the DB
2) Query for secrets which do not have the default key ID or algorithm
3) Decrypts and re-encrypts with new data
4) Wraps all the above in a transaction which asks permission before
   commiting

State secrets in the session state table are deleted. Since they are
short-lived secrets, there is no point in migrating them.

Tested locally.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dmjb dmjb

Labels
user-story
Projects
None yet
Milestone
No milestone
1 participant
@dmjb