Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add tmpf to postgres docker container test #2095

Merged

Conversation

Arsnael
Copy link
Contributor

@Arsnael Arsnael commented Mar 7, 2024

No description provided.

@Arsnael Arsnael self-assigned this Mar 7, 2024
@Arsnael
Copy link
Contributor Author

Arsnael commented Mar 7, 2024

What I noticed, last logs of all hanging forever builds finish by something like this:

08:19:30.425 [INFO ] o.a.j.b.p.DockerPostgresSingleton - 2024-03-06 08:19:30.425 UTC [62] LOG:  checkpoint starting: time
08:23:29.249 [INFO ] o.a.j.b.p.DockerPostgresSingleton - 2024-03-06 08:23:29.249 UTC [62] LOG:  checkpoint complete: wrote 2391 buffers (14.6%); 0 WAL file(s) added, 0 removed, 3 recycled; write=238.594 s, sync=0.002 s, total=238.824 s; sync files=0, longest=0.000 s, average=0.000 s; distance=49321 kB, estimate=49321 kB; lsn=0/493D4F8, redo lsn=0/493D4C0

=> https://www.postgresql.org/docs/current/wal-configuration.html

Honestly, in the build, lots of those checkpoint happen when sutting down the machine and it's fine. The problem is when it kicks before shutdown cause the checkpoint_timeout expired (5 minutes by default... => https://www.postgresql.org/docs/current/runtime-config-wal.html#GUC-CHECKPOINT-TIMEOUT) I checked the times between this one and the previous shutdown checkpoint... 5 minutes.

It seems somehow for some reason that this checkpoint time is making our build hang forever. It seems happening sometimes (but not all the time) in mpt imap tests. I think we play on a very narrow limit here.

So i'm taking the bet of increasing this timeout to 10 minutes instead, let's see.

Also while I'm at it, I added a tmpfs to it...

@Arsnael
Copy link
Contributor Author

Arsnael commented Mar 8, 2024

Still occurred at 10 minutes interval... Not sure to understand what's the difference with the main branch, as it seems occuring in the mpt imap tests anyways and does not even reach the integration tests... I put 1 hour, we don't need those to kick in anyways on PG for testing purposes

@Arsnael
Copy link
Contributor Author

Arsnael commented Mar 8, 2024

Checkpoint has nothing to do with it in fact... what's hanging the mpt tests is on the search, the Optimize AttachmentLoader commit is faulty, could reproduce locally on PostgresUidSearchOnIndexTest for example.

Will investigate

@Arsnael Arsnael force-pushed the postgres-integration-tests-fix-build branch from c938d2d to 0bb2f94 Compare March 8, 2024 09:18
@Arsnael
Copy link
Contributor Author

Arsnael commented Mar 11, 2024

https://ci-builds.apache.org/job/james/job/ApacheJames/job/PR-2095/3/testReport/

I can reproduce two of those 3 locally (fails everytime):

  • PostgresFastViewProjectionHealthCheckIntegrationTest
  • PostgresMailboxSetMethodTest

Those fixes have not been well tested I'm afraid...

@Arsnael Arsnael force-pushed the postgres-integration-tests-fix-build branch from 0bb2f94 to 42e61b0 Compare March 12, 2024 03:04
@Arsnael Arsnael changed the title Postgres integration tests fix build Add tmpf to postgres docker container test Mar 12, 2024
@Arsnael
Copy link
Contributor Author

Arsnael commented Mar 12, 2024

Just reducing to the tmpf addition, which is still nice to have I think

@Arsnael Arsnael merged commit 12da8af into apache:postgresql Mar 12, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants