EKS Upgrade Testing Plan: Step 3: Establish automation connectivity between vets-api & Postgres/RDS

[jennb33]

Vets-API is undergoing an EKS upgrade. As part of the process, it is necessary to conduct comprehensive testing on various components of Vets-API to ensure seamless functionality and performance during the upgrade.

From the EKS Testing Proposal:

As part of the current EKS upgrade process, comprehensive testing on various components of Vets API is essential in order to ensure seamless functionality and performance. Upgrading EKS can introduce changes that affect the behavior of deployed applications. Testing verifies that deployed applications remain compatible with any EKS changes. It ensures that all aspects of Vets API continue to operate as expected, without introducing regressions or unexpected behavior that could disrupt services for users.

Because of future goals to conduct upgrades on a 3-6 month cadence, it is the Product Platform team’s goal to maximize the automation of these upgrade testing processes. By automating these tests, verification of successful upgrades is potentially expedited and the right amount of time for necessary adjustments can be allocated. The Vets API components that require testing are below, along with the potential automation strategies. The objective is to standardize testing methodologies across the platform, thereby eliminating the inconsistency of testing solutions across EKS applications.

This ticket covers step three of testing plan linked in the doc above–Establish automation connectivity between vets-api and Postgres/RDS

NOTE: Make sure that this testing is coordinated with @ph-One in the Infra team prior to testing.

Tasks

• Ensure Vets API can connect to Postgres
• Test in Rails console by accessing records
• Test a GET and PUT operation

Things to consider:

• This will likely happen in an automated (K8) job. How can this be automated?
• Potentially create a cronjob that runs an a rake - either a script or rake task - that hits Redis
• Create Log-Based Alerts in Datadog - specify how notifications need to be received

Success Metrics

• Postgres/RDS is working at 100%
• If the connection fails when being tested, there is an alert sent via the monitor channel

Acceptance Criteria

• Vets API connects to Postgres
• Records are accessed by Test in Rails console
• A GET operation is successful
• A PUT operation is successful
• All of the steps above are AUTOMATED

Validation

• Upon merging to dev and staging, check that https://dev-api.va.gov/v0/status shows the new postgres_up key and value.

[rmtolmach]

Here are some ideas on how to test the Postgres connection in the rails console.

1. Test connection: This should return true:

ActiveRecord::Base.connection.active?

This tests whether the connection to the database is currently open and active.

2. Access records: Would one of these be better at testing this?

Users.nil? - should return false
OR
UserAccount.first - should return a user object like below.

#<UserAccount:0x00007f37403d4d88
 id: "redacted",
 icn: "redacted",
 created_at: Wed, 12 Oct 2022 17:13:08.519719000 UTC +00:00,
 updated_at: Fri, 14 Oct 2022 13:43:32.047816000 UTC +00:00>

3. GET and PUT operations: GET would be tested if we used .first like one of the examples above. For PUT, what about something like this? ChatGPT recommended it and I tested it out in my local Atlas database.

irb(main):014* ProductDirectory::TeamMember.transaction do
irb(main):015*   first = ProductDirectory::TeamMember.first
irb(main):016*   first.update(first_name: 'nothing')
irb(main):017*   raise ActiveRecord::Rollback  # Forces a rollback
irb(main):018> end

I then ran ProductDirectory::TeamMember.first and made sure the first name was not changed. Success.

[LindseySaari]

@rmtolmach On top of adding these checks, we will want to automate calling the endpoint (we were thinking of extending /v0/status or maybe make a new endpoint) but we will want to have automation in place to call this endpoint and report back any failures. I could see a synthetic, cronjob, k8s healthcheck, etc. I could also see breaking this ticket up into two pieces 1) Getting the check in place 2) automating the check and reporting failures

[rmtolmach]

For step 3 in my previous comment, I've tried it in vets-api console. Looks good. And per Lindsey's comment in Slack, I created a user first and then updated it before rolling back. This is because we could be testing this against an empty database.

irb(main):016* UserAccount.transaction do
irb(main):017*   icn = "abc123"
irb(main):018*   UserAccount.create!(icn:)
irb(main):019*   user = UserAccount.find_by(icn:)
irb(main):020*   user.update(icn:"xyz789")
irb(main):021*   raise ActiveRecord::Rollback
irb(main):022> end
2024-04-30 17:28:34.170502 D [94213:12160 log_subscriber.rb:167] ActiveRecord::Base --   TRANSACTION (0.1ms)  BEGIN
2024-04-30 17:28:34.172003 D [94213:12160 log_subscriber.rb:167] ActiveRecord::Base --   UserAccount Exists? (1.7ms)  SELECT 1 AS one FROM "user_accounts" WHERE "user_accounts"."icn" = $1 LIMIT $2  [["icn", "abc123"], ["LIMIT", 1]]
2024-04-30 17:28:34.172750 D [94213:12160 log_subscriber.rb:167] ActiveRecord::Base --   UserAccount Create (0.3ms)  INSERT INTO "user_accounts" ("icn", "created_at", "updated_at") VALUES ($1, $2, $3) RETURNING "id"  [["icn", "abc123"], ["created_at", "2024-04-30 21:28:34.172192"], ["updated_at", "2024-04-30 21:28:34.172192"]]
2024-04-30 17:28:34.173240 D [94213:12160 log_subscriber.rb:167] ActiveRecord::Base --   UserAccount Load (0.1ms)  SELECT "user_accounts".* FROM "user_accounts" WHERE "user_accounts"."icn" = $1 LIMIT $2  [["icn", "abc123"], ["LIMIT", 1]]
2024-04-30 17:28:34.173802 D [94213:12160 log_subscriber.rb:167] ActiveRecord::Base --   UserAccount Exists? (0.1ms)  SELECT 1 AS one FROM "user_accounts" WHERE "user_accounts"."icn" = $1 AND "user_accounts"."id" != $2 LIMIT $3  [["icn", "xyz789"], ["id", "e550e650-4a9f-403b-8042-15e3125a7246"], ["LIMIT", 1]]
2024-04-30 17:28:34.174386 D [94213:12160 log_subscriber.rb:167] ActiveRecord::Base --   UserAccount Update (0.2ms)  UPDATE "user_accounts" SET "icn" = $1, "updated_at" = $2 WHERE "user_accounts"."id" = $3  [["icn", "xyz789"], ["updated_at", "2024-04-30 21:28:34.173934"], ["id", "e550e650-4a9f-403b-8042-15e3125a7246"]]
2024-04-30 17:28:34.174678 D [94213:12160 log_subscriber.rb:167] ActiveRecord::Base --   TRANSACTION (0.1ms)  ROLLBACK
=> nil
irb(main):023>

[rmtolmach]

Rake task that tests the three scenarios for postgres connectivity: file on my branch. Alternatively, here's the diff for easier commenting.

@LindseySaari I'm having trouble understanding how a new endpoint fits in with all of this. I might need to meet with you to get it explained.

[jennb33]

Per @RachalCassity ActiveRecord::Base.connection.active?

[rmtolmach]

• PR up for the rake task: Postgres connectivity status added to v0/status endpoint vets-api#16629

• New ticket to cover the automation around running this rake task: EKS Upgrade Testing Plan: Step 3, PART II: Establish automation connectivity between vets-api & Postgres/RDS #82296

[rmtolmach]

I made some changes to the PR:

I abandoned the rake task and moved the logic into a controller and returned true or false, so when we hit v0/status, it returns true/false.

• then make a synthetic to hit the endpoint. We want to test this from inside the cluster and from the outside. This is done in the next ticket (EKS Upgrade Testing Plan: Step 3, PART II: Establish automation connectivity between vets-api & Postgres/RDS #82296)

I added a validation step that can be done after merge into dev.

[rmtolmach]

Validation:
✅ https://dev-api.va.gov/v0/status

{"git_revision":"4707c1ad030687f440a91f31df4b7eda02bc6877","db_url":null,"postgres_up":true}

Per Rachal's suggestion, I changes the logging to be at the info level instead of error. She suggested that in my original PR, but I misunderstood the ask, so I changed it in a follow-up PR: department-of-veterans-affairs/vets-api#16683 and that has been merged.

Closing.