JAMES-2586 Fix some jmap postgres integration tests #2048

hungphan227 · 2024-02-25T08:19:37Z

No description provided.

...ration-tests/src/test/java/org/apache/james/jmap/rfc8621/postgres/PostgresThreadGetTest.java

.../postgres/src/main/java/org/apache/james/mailbox/postgres/mail/PostgresAttachmentModule.java

...-common/src/main/scala/org/apache/james/jmap/rfc8621/contract/EmailQueryMethodContract.scala

...postgres/src/main/java/org/apache/james/mailbox/postgres/mail/dao/PostgresAttachmentDAO.java

quantranhong1999 · 2024-02-29T04:49:51Z

org.apache.james.events.PostgresEventDeadLettersTest

Related?

Arsnael · 2024-03-04T07:53:28Z

Please rebase

vttranlina · 2024-03-05T10:32:14Z

...x/postgres/src/main/java/org/apache/james/mailbox/postgres/mail/PostgresMessageIdMapper.java

                }
                return Mono.just(messageBuilder.build());
            }, ReactorUtils.DEFAULT_CONCURRENCY);
    }

+    private Mono<SimpleMailboxMessage> retrieveFullMessage(SimpleMailboxMessage.Builder messageBuilder, Record record) {


I see a lot of methods were written for adding attachment metadata to SimpleMailboxMessage.Builder.
it is similar to what we did in AttachmentLoader

Can we reuse it?

I remember that AttachmentLoader's method does not fit this case

let me try it a bit

caught

Current AttachmentLoader get each by each MessageAttachmentMetadata by id

Your new code: get list MessageAttachmentMetadata by list id 👍

I have a refactor for that 437550f
can you look it

great. tks. could you push it to this PR?

I don't have permission, you can fetch my code, and cherry-pick this commit

FYI that commit is what is making tests failing/hanging forever on imap mpt tests

...postgres/src/main/java/org/apache/james/mailbox/postgres/mail/dao/PostgresAttachmentDAO.java

vttranlina · 2024-03-06T04:19:30Z

Related to the ci failed

bodyStructureShouldSupportSpecificHeaders(GuiceJamesServer) – org.apache.james.jmap.rfc8621.postgres.PostgresEmailGetMethodTest

JSON documents are different:
Different value found in node "methodResponses[0][1].list[0].bodyStructure.blobId", expected: <"1_1"> but was: <"018e11c3-8f00-7d96-a046-4f8beef18834_1">.

I see we get the same in the Distributed test (master branch)

It looks like comes from test cases, not related to Postgres or not
//
I'm investigating it

vttranlina · 2024-03-06T04:53:03Z

Related to the ci failed bodyStructureShouldSupportSpecificHeaders(GuiceJamesServer)

we can cherry-pick fixup commit: #2093

Arsnael · 2024-03-07T08:29:13Z

03:25:25.251 [INFO ] o.a.j.b.p.DockerPostgresSingleton - 2024-03-07 03:25:25.250 UTC [62] LOG:  checkpoint starting: time
03:27:57.599 [INFO ] o.a.j.b.p.DockerPostgresSingleton - 2024-03-07 03:27:57.598 UTC [62] LOG:  checkpoint complete: wrote 1524 buffers (9.3%); 0 WAL file(s) added, 0 removed, 1 recycled; write=152.270 s, sync=0.001 s, total=152.348 s; sync files=0, longest=0.000 s, average=0.000 s; distance=16207 kB, estimate=16207 kB; lsn=0/28E6CE0, redo lsn=0/28E6CA8

Hanging again, same point... do we have an issue here in our pg test with the docker postgres signleton extension?

we should investigate...

I'm suspecting something being potentially unstable on our side as all the timeout builds have occured on different workers in the last few days

quantranhong1999 · 2024-03-07T08:38:39Z

I'm suspecting something being potentially unstable on our side as all the timeout builds have occured on different workers in the last few days

Any chance it is related to this change:
https://github.com/apache/james-project/pull/2072/files

...

Arsnael · 2024-03-07T09:07:08Z

No I don't think so... I think I have a clue, and maybe we are a bit unlucky on some of our builds... let me try something and open an other PR, I noticed a pattern when such thing happens

Arsnael · 2024-03-07T09:39:10Z

@quantranhong1999 => #2095

My 50 cents, let's see :)

Arsnael · 2024-03-08T09:21:47Z

@hungphan227 @vttranlina I'm going on my PR to test back without the last commit, could not fix it.

Feel free to fix it yourselves if you want it back

For example locally you can try to run UidSearchOnIndex, you will see quickly it's hanging forever

vttranlina · 2024-03-11T03:58:37Z

fixup! JAMES-2586 Optimize AttachmentLoader

Fixed 144ac90

Reason: Hanging when Collections is empty

Arsnael · 2024-03-11T04:35:19Z

Thanks.

Can you test guys locally on those tests and make sure it passes with those fixes? => https://ci-builds.apache.org/job/james/job/ApacheJames/job/PR-2095/3/testReport/

vttranlina · 2024-03-11T04:54:44Z

Thanks.

Can you test guys locally on those tests and make sure it passes with those fixes? => https://ci-builds.apache.org/job/james/job/ApacheJames/job/PR-2095/3/testReport/

webSocketShouldPushNewMessageWhenChangeSubscriptionOfMailbox -> failed on my local. Related it, I discussed it with @hungphan227 on last Thursday, it failed because the test suite code, was not related to Postgres -> should update the test code
checkShouldTurnFromDegradedToHealthyAfterMoreReadsOnAMissedProjection -> failed
shouldFailWithCannotCalculateChangesWhenSingleChangeIsTooLarge -> passed

Arsnael · 2024-03-12T02:53:12Z

11:29:37,710 [INFO] Apache James :: Server :: JMAP RFC-8621 :: Postgres Integration Testing FAILURE [46:53 min]

46 mins... Maybe unstable CI or maybe not. Will restart the build to see

…st, PostgresMailboxSetMethodTest

…etTest

- get the list replaced to get each by each

Arsnael · 2024-03-12T07:16:43Z

Build was green (finally)

I squashed and rebased

Arsnael · 2024-03-13T02:18:00Z

The JMAP integration test part seems to take often 40 ~ 50 minutes to complete which is too long (we consider less than 30 minutes ok)

@hungphan227 Can we investigate again?

quantranhong1999 reviewed Feb 26, 2024

View reviewed changes

hungphan227 mentioned this pull request Feb 26, 2024

[Postgres] Fix some JMAP postgres integration tests linagora/james-project#5077

Closed

vttranlina reviewed Feb 26, 2024

View reviewed changes

...postgres/src/main/java/org/apache/james/mailbox/postgres/mail/dao/PostgresAttachmentDAO.java Outdated Show resolved Hide resolved

hungphan227 force-pushed the JAMES-2586-Fix-some-JMAP-postgres-integration-tests branch 2 times, most recently from ec122f5 to 857c591 Compare March 4, 2024 10:01

Arsnael approved these changes Mar 5, 2024

View reviewed changes

vttranlina reviewed Mar 5, 2024

View reviewed changes

hungphan227 force-pushed the JAMES-2586-Fix-some-JMAP-postgres-integration-tests branch from 857c591 to 5a716f0 Compare March 6, 2024 07:58

vttranlina approved these changes Mar 6, 2024

View reviewed changes

Arsnael approved these changes Mar 6, 2024

View reviewed changes

vttranlina mentioned this pull request Mar 11, 2024

JAMES-2586 [Postgres] FIXUP when query with IN - should pre-check collection size #2103

Merged

hungphan227 force-pushed the JAMES-2586-Fix-some-JMAP-postgres-integration-tests branch from ac1309f to b2ead22 Compare March 11, 2024 07:13

hung phan and others added 4 commits March 12, 2024 14:15

JAMES-2586 Fix PostgresEmailGetMethodTest, PostgresEmailQueryMethodTe…

55a99a1

…st, PostgresMailboxSetMethodTest

JAMES-2586 Fix PostgresPushSubscriptionSetMethodTest, PostgresThreadG…

c5641f1

…etTest

JAMES-2586 Optimize AttachmentLoader

1b16d50

- get the list replaced to get each by each

JAMES-2586 Replace drop by truncate in PostgresMessageFastViewProjection

ec60c86

Arsnael force-pushed the JAMES-2586-Fix-some-JMAP-postgres-integration-tests branch from b2ead22 to ec60c86 Compare March 12, 2024 07:16

Arsnael mentioned this pull request Mar 12, 2024

JAMES-2586 Avoid sorting PG messages #2092

Merged

hungphan227 mentioned this pull request Mar 13, 2024

JAMES-2586 Fix PostgresEmailGetMethodTest, PostgresEmailQueryMethodTest, PostgresMailboxSetMethodTest #2108

Merged

hungphan227 closed this Mar 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

JAMES-2586 Fix some jmap postgres integration tests #2048

JAMES-2586 Fix some jmap postgres integration tests #2048

hungphan227 commented Feb 25, 2024

quantranhong1999 commented Feb 29, 2024

Arsnael commented Mar 4, 2024

vttranlina Mar 5, 2024

hungphan227 Mar 6, 2024 •

edited

vttranlina Mar 6, 2024

vttranlina Mar 6, 2024

hungphan227 Mar 6, 2024

vttranlina Mar 6, 2024

Arsnael Mar 8, 2024 •

edited

vttranlina commented Mar 6, 2024

vttranlina commented Mar 6, 2024

Arsnael commented Mar 7, 2024 •

edited

quantranhong1999 commented Mar 7, 2024

Arsnael commented Mar 7, 2024

Arsnael commented Mar 7, 2024

Arsnael commented Mar 8, 2024

vttranlina commented Mar 11, 2024

Arsnael commented Mar 11, 2024

vttranlina commented Mar 11, 2024

Arsnael commented Mar 12, 2024

Arsnael commented Mar 12, 2024

Arsnael commented Mar 13, 2024

JAMES-2586 Fix some jmap postgres integration tests #2048

JAMES-2586 Fix some jmap postgres integration tests #2048

Conversation

hungphan227 commented Feb 25, 2024

quantranhong1999 commented Feb 29, 2024

Arsnael commented Mar 4, 2024

vttranlina Mar 5, 2024

Choose a reason for hiding this comment

hungphan227 Mar 6, 2024 • edited

Choose a reason for hiding this comment

vttranlina Mar 6, 2024

Choose a reason for hiding this comment

vttranlina Mar 6, 2024

Choose a reason for hiding this comment

hungphan227 Mar 6, 2024

Choose a reason for hiding this comment

vttranlina Mar 6, 2024

Choose a reason for hiding this comment

Arsnael Mar 8, 2024 • edited

Choose a reason for hiding this comment

vttranlina commented Mar 6, 2024

vttranlina commented Mar 6, 2024

Arsnael commented Mar 7, 2024 • edited

quantranhong1999 commented Mar 7, 2024

Arsnael commented Mar 7, 2024

Arsnael commented Mar 7, 2024

Arsnael commented Mar 8, 2024

vttranlina commented Mar 11, 2024

Arsnael commented Mar 11, 2024

vttranlina commented Mar 11, 2024

Arsnael commented Mar 12, 2024

Arsnael commented Mar 12, 2024

Arsnael commented Mar 13, 2024

hungphan227 Mar 6, 2024 •

edited

Arsnael Mar 8, 2024 •

edited

Arsnael commented Mar 7, 2024 •

edited