feat: Encrypted fields rework #24966

benjackwhite · 2024-09-13T15:44:35Z

Problem

Ensures the fields are encrypted and not just removed from payloads.

Changes

Adds the encrypted_inputs value
Changes the logic to write to them and read back but redacted
Remove need for secret key
Works in the plugin server
New secret values added to deployments - https://github.com/PostHog/charts/pull/1828

Follow up

Migrate all fields to the new format
Also migrate data warehouse fields

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

Does this work well for both Cloud and self-hosted?

How did you test this code?

github-actions · 2024-09-13T15:55:34Z

Size Change: 0 B

Total Size: 1.1 MB

ℹ️ View Unchanged

Filename	Size
`frontend/dist/toolbar.js`	1.1 MB

_{compressed-size-action}

posthog-bot · 2024-09-16T08:19:33Z

📸 UI snapshots have been updated

1 snapshot changes in total. 0 added, 1 modified, 0 deleted:

chromium: 0 added, 1 modified, 0 deleted (diff for shard 2)
webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

…g into feat/cdp-field-encryption

posthog-bot · 2024-09-16T12:32:37Z

📸 UI snapshots have been updated

1 snapshot changes in total. 0 added, 1 modified, 0 deleted:

chromium: 0 added, 1 modified, 0 deleted (diff for shard 2)
webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

…g into feat/cdp-field-encryption

posthog-bot · 2024-09-17T10:39:24Z

📸 UI snapshots have been updated

1 snapshot changes in total. 0 added, 1 modified, 0 deleted:

chromium: 0 added, 1 modified, 0 deleted (diff for shard 1)
webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

# Conflicts: # .github/workflows/ci-e2e.yml

# Conflicts: # latest_migrations.manifest

…g into feat/cdp-field-encryption

mariusandra

Playing around locally. I had

no keys set: print(inputs.secret) in a function worked
set a key for the plugin server only: print(inputs.secret) returned null
set the same key for django: things started crashing

It appears django wasn't able to decrypt the previous encrypted_inputs field, and instead of returing None or a dict, it returned the encrypted string. Then followup code that expects a dict crashed.

I committed a patch to just return None instead in this case. Then everything worked and I could print the secret again (showed up as redacted).

mariusandra · 2024-09-18T10:12:07Z

.github/workflows/ci-e2e.yml

@@ -191,6 +191,7 @@ jobs:
                  GITHUB_ACTION_RUN_URL="${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
                  CELERY_METRICS_PORT=8999
                  CLOUD_DEPLOYMENT=E2E
+                  ENCRYPTION_SALT_KEYS=9bf7f7c13cf2148a23d6b07557f95cb7


ProTip™️: deadbeef is legit hex, as is 00beef0000beef0000beef0000beef00 if you want something that looks less like a real key

# Conflicts: # plugin-server/package.json # plugin-server/src/config/config.ts # plugin-server/src/types.ts

posthog-bot · 2024-09-19T08:50:44Z

📸 UI snapshots have been updated

1 snapshot changes in total. 0 added, 1 modified, 0 deleted:

chromium: 0 added, 1 modified, 0 deleted (diff for shard 2)
webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

posthog-bot · 2024-09-19T09:10:08Z

📸 UI snapshots have been updated

1 snapshot changes in total. 0 added, 1 modified, 0 deleted:

chromium: 0 added, 1 modified, 0 deleted (diff for shard 2)
webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

Started reworking encrypted fields

1ff961c

benjackwhite and others added 3 commits September 16, 2024 10:00

Fixes

ebe9350

Fix up encrypted fields usage

f218c31

Update UI snapshots for chromium (2)

4148934

benjackwhite added 9 commits September 16, 2024 10:20

Fix up inputs

0bd63a0

Merge branch 'feat/cdp-field-encryption' of github.com:PostHog/postho…

651bb64

…g into feat/cdp-field-encryption

Started reworking secrets

746f955

Fixes

9bce281

Fix tests

d2f7728

Fixes

9791f0e

Added encryption helper

fef3d91

Added decrypting of fields to the function manager

a0f638d

Fix

9ab2247

abhi12299 mentioned this pull request Sep 16, 2024

fix(insights): truncated charts in firefox #24990

Open

benjackwhite and others added 4 commits September 16, 2024 14:21

Fixes

4b666bf

Fixes

94b8dd2

Fixes

bb86b3b

Update UI snapshots for chromium (2)

c032f9d

benjackwhite added 3 commits September 16, 2024 14:33

Fix tests

1e3bfb6

Added static collection helper

cb9d604

Merge branch 'feat/cdp-field-encryption' of github.com:PostHog/postho…

1d515d6

…g into feat/cdp-field-encryption

benjackwhite marked this pull request as ready for review September 16, 2024 12:41

benjackwhite requested review from mariusandra, neilkakkar and EDsCODE September 16, 2024 12:41

benjackwhite changed the title ~~feat(cdp): Encrypted cdp fields~~ feat: Encrypted fields rework Sep 16, 2024

Update UI snapshots for chromium (1)

00b99b0

Update UI snapshots for chromium (1)

3ea4ebc

benjackwhite and others added 14 commits September 17, 2024 12:41

Add migration

00c17ad

Fix

6cf1637

Added data warehouse migrations

6c2c6cb

Final posthog side things

db30b2c

Added encrypted fields

3e91a35

Merge branch 'feat/cdp-test-e2e' into feat/cdp-field-encryption

495ca33

# Conflicts: # .github/workflows/ci-e2e.yml

Log them out

d2db0c4

Fix

45e5ea2

Fix

8159fcb

Merge branch 'master' into feat/cdp-field-encryption

4ca3a19

# Conflicts: # latest_migrations.manifest

Merge branch 'master' into feat/cdp-field-encryption

ca77d7a

Merge branch 'master' into feat/cdp-field-encryption

1be6c81

only return if dicts

f4eaaa6

Merge branch 'feat/cdp-field-encryption' of github.com:PostHog/postho…

1b2e2ae

…g into feat/cdp-field-encryption

mariusandra approved these changes Sep 18, 2024

View reviewed changes

benjackwhite and others added 5 commits September 18, 2024 13:19

Fix

a7b4ddc

Fix up key

95e8e9d

Merge branch 'master' into feat/cdp-field-encryption

25ca3ad

# Conflicts: # plugin-server/package.json # plugin-server/src/config/config.ts # plugin-server/src/types.ts

Fix

1e2b784

Update UI snapshots for chromium (2)

273ebd7

Update UI snapshots for chromium (2)

ca8c924

benjackwhite added 2 commits September 19, 2024 12:39

Fixes

d661b4e

Merge branch 'master' into feat/cdp-field-encryption

9133dce

benjackwhite merged commit ad32faf into master Sep 19, 2024
93 checks passed

benjackwhite deleted the feat/cdp-field-encryption branch September 19, 2024 12:32

Kvintus mentioned this pull request Oct 4, 2024

Plugin Server Fails to Start on Fresh Self-Hosted Docker Install (Ubuntu 24.04) #25373

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Encrypted fields rework #24966

feat: Encrypted fields rework #24966

benjackwhite commented Sep 13, 2024 •

edited

Loading

github-actions bot commented Sep 13, 2024 •

edited

Loading

posthog-bot commented Sep 16, 2024

posthog-bot commented Sep 16, 2024

posthog-bot commented Sep 17, 2024

mariusandra left a comment •

edited

Loading

mariusandra Sep 18, 2024

benjackwhite Sep 18, 2024

posthog-bot commented Sep 19, 2024

posthog-bot commented Sep 19, 2024

feat: Encrypted fields rework #24966

feat: Encrypted fields rework #24966

Conversation

benjackwhite commented Sep 13, 2024 • edited Loading

Problem

Changes

Follow up

Does this work well for both Cloud and self-hosted?

How did you test this code?

github-actions bot commented Sep 13, 2024 • edited Loading

posthog-bot commented Sep 16, 2024

📸 UI snapshots have been updated

posthog-bot commented Sep 16, 2024

📸 UI snapshots have been updated

posthog-bot commented Sep 17, 2024

📸 UI snapshots have been updated

mariusandra left a comment • edited Loading

Choose a reason for hiding this comment

mariusandra Sep 18, 2024

Choose a reason for hiding this comment

benjackwhite Sep 18, 2024

Choose a reason for hiding this comment

posthog-bot commented Sep 19, 2024

📸 UI snapshots have been updated

posthog-bot commented Sep 19, 2024

📸 UI snapshots have been updated

benjackwhite commented Sep 13, 2024 •

edited

Loading

github-actions bot commented Sep 13, 2024 •

edited

Loading

mariusandra left a comment •

edited

Loading