Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DPE-3473] Stabilise SST and network cut tests #385

Merged
merged 4 commits into from
Feb 7, 2024
Merged

[DPE-3473] Stabilise SST and network cut tests #385

merged 4 commits into from
Feb 7, 2024

Conversation

marceloneppel
Copy link
Member

@marceloneppel marceloneppel commented Feb 2, 2024 •
edited

Issue

The SST test sometimes fails with the following error:

AssertionError: secondary not up to date with the cluster after restarting.

Also, the network cut test constantly fails with the following error:

AssertionError: Connection is not possible after network restore

Solution

Add more information to logs and more logs to tests (to help identify the causes of the failures).

For the SST test:

  • Reinitialised the replica (former primary) data directory if it stopped streaming data from the new primary due to WAL files missing in the primary.

For the network cut test:

  • Added a retry to the attempt to connect to the database after the network connection was restored (DPE-3489 was created to investigate why it was failing).

More tests (like the scaling to zero units test) will be fixed in future PRs.

@marceloneppel
Stabilise network cut test
68ad63e 
Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
@marceloneppel
Fix forceful restart without data and transaction logs test
12b23f1 
Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
@marceloneppel
Merge remote-tracking branch 'origin/main' into dpe-3473-stabilise-ha…
eb20f81 
…-tests

Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
@marceloneppel
Minor fixes + add unit tests
c7543a8 
Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
@marceloneppel marceloneppel changed the title [DPE-3473] Stabilise HA tests [DPE-3473] Stabilise SST and network cut tests Feb 6, 2024
@marceloneppel marceloneppel marked this pull request as ready for review February 6, 2024 22:20
taurus-forever
taurus-forever approved these changes Feb 7, 2024
View reviewed changes
Copy link
Contributor

@taurus-forever taurus-forever left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a great test improvement. THANK YOU!

@marceloneppel marceloneppel merged commit 098a4c2 into main Feb 7, 2024
41 checks passed
@marceloneppel marceloneppel deleted the dpe-3473-stabilise-ha-tests branch February 7, 2024 18:04
BON4 pushed a commit to BON4/postgresql-k8s-operator that referenced this pull request May 20, 2024
@marceloneppel
[DPE-3473] Stabilise SST and network cut tests (canonical#385)
a497159 
* Stabilise network cut test

Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>

* Fix forceful restart without data and transaction logs test

Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>

* Minor fixes + add unit tests

Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>

---------

Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@taurus-forever taurus-forever taurus-forever approved these changes

@dragomirp dragomirp dragomirp approved these changes

Assignees
No one assigned
Labels
Libraries: OK
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@marceloneppel @taurus-forever @dragomirp