Migrate Postgres to Neon.tech

[gerhard]

This is a follow-up to:

• Bump deps to latest #490

https://changelog-2024-01-12.fly.dev/ is configured to use https://console.neon.tech/app/projects/orange-sound-86604986

Initial observations

1/3. P99 latency for /feed increased by 3x - 3s vs 1s

2/3. Ecto SSL config doesn't seem to be working with :verify_peer

See the this commit for more details. FTR:

• chore: Enable TLS connection to postgres on newer OTP versions kitsteam/wordcharts#4

We figured it out with @brendan-stephens via #492 (comment)

3/3. We are doing 70+ SELECTS when serving /

This seems a lot, but maybe it's necessary. Since each SELECT adds an extra 2-10ms due to the network latency, it is reasonable to expect an extra 500ms latency across 70+ SELECT statements.

See changelog-2024-01-12.fly.dev/ vs changelog-2022-03-13.fly.dev/. Now addressed in #492 (comment)

I expect other pages such as /feed which use 473 SELECT statements to result in ever slower responses. Initial observations suggest 4.7s. vs the typical 1.7s.

Next steps

• @gerhard to reach out to Neon Support & figure out what he's doing wrong with :verify_peer
• @gerhard to merge Send all app traces to Honeycomb #496 and rebase this on top so that we can see all traces on changelog-2022-03-13 app instance
• @jerodsanto, @adamstac & @gerhard to decide if the extra latency is worth the migration
• Maybe figure out if we want to fix the podcast & post URLs. They are all defaulting to changelog.com which makes comparing the two origins side-by-side difficult:
• Get :verify_peer working in ssl_opts
• Deploy new changelog-2024-01-12 app instance
• Prepare Fastly config (remember the failed v176!)
• Stop current instance
• Restore db on new instance & ensure everything works as expected
• Promote Fastly config
• Ensure that everything looks good in Honeycomb.io & nothing triggers

After a few days, when we confirm that everything looks right, delete changelog-2022-03-13 app instance + changelog-postgres-2023-07-31 & party 🎉

[gerhard]

I have decided to hook up the 1Password Service Account part of this so that we only need to configure 1 secret in the new app, OP_SERVICE_ACCOUNT_TOKEN, and then op CLI configures all app secrets just-in-time during boot.

The biggest benefit is that secrets will be version controlled, in env.op, same as we do in nightly.

As I'm going this, I would like to double-check that we still use these. This is what I think we need:

• AWS_ACCESS_KEY_ID
• AWS_API_HOST
• AWS_SECRET_ACCESS_KEY
• BUFFER_TOKEN
• CM_API_TOKEN
• CM_SMTP_TOKEN
• DATABASE_URL
• DB_HOST
• DB_PASS
• DB_USER
• FASTLY_TOKEN
• GITHUB_API_TOKEN
• GITHUB_CLIENT_ID
• GITHUB_CLIENT_SECRET
• HONEYCOMB_API_KEY
• MASTODON_API_TOKEN
• MASTODON_CLIENT_ID
• MASTODON_CLIENT_SECRET
• NOTION_API_TOKEN
• PLUSPLUS_SLUG
• R2_ACCESS_KEY_ID
• R2_API_HOST
• R2_SECRET_ACCESS_KEY
• SECRET_KEY_BASE
• SHOPIFY_API_KEY
• SHOPIFY_API_PASSWORD
• SIGNING_SALT
• SLACK_APP_API_TOKEN
• SLACK_INVITE_API_TOKEN
• TURNSTILE_SECRET_KEY
• TWITTER_CONSUMER_KEY
• TWITTER_CONSUMER_SECRET
• TYPESENSE_API_KEY
• TYPESENSE_URL

Did I cross out all secrets that we no longer use @jerodsanto?

[jerodsanto]

Looks correct.

[gerhard]

Thanks for confirming @jerodsanto!

I now have all of this wired up & working as https://changelog-2023-12-17.fly.dev/ . See the PR description for initial observations & next steps.

Leaving something here for Neon Support:

---

I am unable to make https://neon.tech/docs/guides/elixir-ecto#configure-ecto work for our Elixir application.

If I use the following config with our credentials:

config :friends, Friends.Repo,
  database: "friends",
  username: "alex",
  password: "AbC123dEf",
  hostname: "ep-cool-darkness-123456.us-west-2.aws.neon.tech",
  ssl: true,
  ssl_opts: [
    server_name_indication: 'ep-cool-darkness-123456.us-west-2.aws.neon.tech',
    verify: :verify_none
  ]

I get the following errors:

** (Postgrex.Error) ERROR 26000 (invalid_sql_statement_name) prepared statement "ecto_1922" does not exist

If I follow the https://neon.tech/docs/guides/elixir-ecto#configure-ecto documentation further and configure verify: :verify_peer:

  ssl_opts: [
    cacerts: :public_key.cacerts_get(), # available since OTP26
    verify: :verify_peer,
    server_name_indication: String.to_charlist(System.get_env("DB_HOST", "db")),
    customize_hostname_check: [
      match_fun: :public_key.pkix_verify_hostname_match_fun(:https)
    ]
  ],

  ssl_opts: [
    verify: :verify_peer,
    cacerts: :public_key.cacerts_get(), # available since OTP26
    versions: [:"tlsv1.3"],
    depth: 3,
    server_name_indication: String.to_charlist(System.get_env("DB_HOST", "db")),
    customize_hostname_check: [
      match_fun: :public_key.pkix_verify_hostname_match_fun(:https)
    ]
  ],

Ecto is not even able to connect to Neon:

17:41:53.361 [notice] Application changelog exited: Changelog.Application.start(:normal, []) returned an error: shutdown: failed to start child: Changelog.EpisodeTracker
    ** (EXIT) an exception was raised:
        ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 2955ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:

  1. Ensuring your database is available and that you can connect to it
  2. Tracking down slow queries and making sure they are running fast enough
  3. Increasing the pool_size (although this increases resource consumption)
  4. Allowing requests to wait longer by increasing :queue_target and :queue_interval

See DBConnection.start_link/2 for more information

            (ecto_sql 3.10.2) lib/ecto/adapters/sql.ex:1047: Ecto.Adapters.SQL.raise_sql_call_error/1
            (ecto_sql 3.10.2) lib/ecto/adapters/sql.ex:945: Ecto.Adapters.SQL.execute/6
            (ecto 3.10.3) lib/ecto/repo/queryable.ex:229: Ecto.Repo.Queryable.execute/4
            (ecto 3.10.3) lib/ecto/repo/queryable.ex:19: Ecto.Repo.Queryable.all/3
            (changelog 0.0.1) lib/changelog/schema/episode/episode.ex:421: Changelog.Episode.flatten_for_filtering/1
            (changelog 0.0.1) lib/changelog/episode_tracker.ex:130: Changelog.EpisodeTracker.refresh_episodes/0
            (changelog 0.0.1) lib/changelog/episode_tracker.ex:57: Changelog.EpisodeTracker.init/1
            (stdlib 5.2) gen_server.erl:980: :gen_server.init_it/2

This is the only config option that works - in combination with specifying the endpoint id in the password field

  ssl_opts: [ verify: :verify_none ],

While the above workaround works, I am not comfortable skipping remote peer verification in production.

What am I doing wrong?

[gerhard]

Leaving this fun gotcha here:

Screen.Recording.2023-12-30.at.17.13.44.mp4

[jerodsanto]

Rebasing master should reduce the number of SELECTs on the home page and podcast pages. It should also fix the URLs on podcasts, but I haven't done the posts URL one yet...

[gerhard]

That was helpful @jerodsanto!

I just rebased on top of master, re-deployed & also re-imported the db. This is what I'm seeing now:

---

https://changelog-2022-03-13.fly.dev/ uses 15 queries & resolves in 134.6ms

---

https://changelog-2023-12-17.fly.dev/ uses 15 queries & resolves in 151.9ms, meaning .1x slower, which is a 20-30x improvement from what we had when this PR was opened.

---

That is a sweet improvement @jerodsanto 💪

I am going to compare the /feed URL next.

[gerhard]

https://changelog-2022-03-13.fly.dev/feed typically resolves in ~1s & uses 477 SELECT statements.

https://changelog-2023-12-17.fly.dev/feed resolves in ~2-3s & uses 477 SELECT statements.

[jerodsanto]

Worth noting that /feed is not actually served from the app in production because Fastly fetches it from R2. Same goes for all feed endpoints.

[gerhard]

Good point! 830k feed requests are served from R2, and only 27k from Fly.io in the last 7 days

---

Also, in the last 7 days, /feed/ is the 3rd most requested URL from Fly.io

[gerhard]

ssl_opts with :verify_peer are still failing as initially documented:

[gerhard]

Thanks for the pairing session @brendan-stephens 💪

Links:

• Secure SSL defaults elixir-ecto/myxql#61 (comment)
• https://github.com/elixir-mint/castore/blob/main/lib/castore.ex
• https://community.neon.tech/t/guide-on-connecting-via-ecto/75
• https://github.com/ankane/pghero

[gerhard]

https://changelog.com Postgres is now on Neon.tech.

There were a few issues, but nothing major: https://ui.honeycomb.io/changelog/datasets/fastly/result/Eb7gHnHDygg

---

P99 latency is 6% higher - 1.57s vs 1.48s - which is hardly note-worthy: https://ui.honeycomb.io/changelog/datasets/fastly/result/cSinitLo7TX

[gerhard]

This looks good to me: last 30 minutes compared to a day before