Federation tests replication round1 - demonstrate absent replication of comment deletes #3657

RocketDerp · 2023-07-18T12:54:42Z

This is an urgent test addition to highlight the problem with comment deletes not replicating when a remote-server creates the comment. The community home instance has no code to replicate delete of comment to all the downstream subscribed instances. In these tests, Gamma serves as an example of the downstream servers subscribed who are not getting the delete in 0.18.2 version.

The intention here is to put more developer eyes on #3625 as a design issue. I am still working to devise additional tests to see if the problem goes beyond comment deletes.

… downstream from community home

…n, this is currently failing because gamma does not get the report

…bout the LEMMY_DATABASE_URL format (LemmyNet#3651)

* Add http cache for webfingers * Remove the outgoing cache middleware & adjust the cache headers directive * Use 1h & 3day cache header * Update routes and adjust the cache headers location * revert apub caching --------- Co-authored-by: Dessalines <dessalines@users.noreply.github.com> Co-authored-by: Felix Ableitner <me@nutomic.com>

Nutomic · 2023-07-19T14:18:35Z

api_tests/src/comment.spec.ts

+  // This is testing replication from remote-home-remote (alpha-beta-gamma)
+  let gammaComment = (
+    await resolveComment(gamma, commentRes.comment_view.comment)
+  ).comment;


This isnt really testing replication. If the comment failed to be federated, resolve will fetch it instead. You could instead use resolvePost and then getComments for that post to check if its included there. Or add a new getComments variant which retrieves all newest comments for the whole instance.

Nutomic · 2023-07-19T14:45:51Z

api_tests/src/comment.spec.ts

+      console.log("gamma has deleted comment", gammaCommentRes.comment.comment);
+    }
+  }
+  expect(gammaCommentRes.error).toBe("couldnt_find_object");


Okay turns out its like I mentioned in my other comment, the problem is in activity_lists.rs. Removing the line PersonInboxActivities::Delete fixes this test. So it enters the wrong variant which doesnt forward it to followers. Have to think about the proper way to fix it, probably some rewrite of that file is necessary.

FYI the way to debug this is by looking at the Lemmy logs in less /tmp/lemmy_beta.out etc, and adding dbg! statements in the Rust code to see what it is doing.

Nutomic · 2023-07-19T14:51:00Z

api_tests/src/comment.spec.ts

+
+  // This is testing replication from remote-home-remote(s) (alpha-beta-gamma)
+  // The sitaution could be that a remote moderator is on Gamma and needs to see the report in doing their moderator duty.
+  let gammaReports = await listCommentReports(gamma);


This probably fails because report is not announcable, so it will only go to the community instance but not anywhere else.

Nutomic · 2023-07-20T10:53:15Z

I pushed a commit to your branch which fixes the federation of deletes. You need to run git pull to get it locally. Please remove the changes to "Report a comment" test so that we can go ahead and merge this. You also need to run prettier on the ts files.

Nutomic · 2023-07-20T10:54:17Z

api_tests/run-federation-test.sh

@@ -13,7 +13,7 @@ popd
 yarn
 yarn api-test || true

-killall lemmy_server
+killall -s1 lemmy_server


Without this there were still lemmy processes left running after tests completed.

This reverts commit 7dfb6ee.

…situation, this is currently failing because gamma does not get the report" This reverts commit 7bd3b20.

Nutomic · 2023-07-21T10:30:38Z

Here is the prettier issue for showing a diff on check. No progress in 4 years...

That tmknom is irrelevant, its just the person who built a dockerfile for prettier which we are using. It shouldnt have any changes compared to other ways of running prettier. I run npx prettier -w {path}, pushed a commit with that to your branch but somehow formatting is still failing. cc @dessalines

Nutomic · 2023-07-27T09:53:20Z

There was another failure in api tests which I had to fix.

…of comment deletes (#3657) * more robust test of unlike a comment, confirm replication to instance downstream from community home * more robust 'delete a comment' test, confirm replication * Far more robust "Report a comment" test. Many comments about situation, this is currently failing because gamma does not get the report * typo and actually have Gamma comment check use gamma, not alpha * prepare-drone-federation-test.sh has some more echo output and note about the LEMMY_DATABASE_URL format (#3651) * Add http cache for webfingers (#3317) * Add http cache for webfingers * Remove the outgoing cache middleware & adjust the cache headers directive * Use 1h & 3day cache header * Update routes and adjust the cache headers location * revert apub caching --------- Co-authored-by: Dessalines <dessalines@users.noreply.github.com> Co-authored-by: Felix Ableitner <me@nutomic.com> * Rewrite activity lists to fix delete federation (fixes #3625) * Revert "typo and actually have Gamma comment check use gamma, not alpha" This reverts commit 7dfb6ee. * Revert "Far more robust "Report a comment" test. Many comments about situation, this is currently failing because gamma does not get the report" This reverts commit 7bd3b20. * prettier TypeScript * revised comments, as ResolveObject isn't using routine replication * fmt * fix api tests * remove comment --------- Co-authored-by: cetra3 <cetra3@hotmail.com> Co-authored-by: Dessalines <dessalines@users.noreply.github.com> Co-authored-by: Felix Ableitner <me@nutomic.com>

RocketDerp added 2 commits July 18, 2023 05:51

more robust test of unlike a comment, confirm replication to instance…

fb7df85

… downstream from community home

more robust 'delete a comment' test, confirm replication

83cdac2

RocketDerp requested review from Nutomic, dessalines and phiresky as code owners July 18, 2023 12:54

Far more robust "Report a comment" test. Many comments about situatio…

7bd3b20

…n, this is currently failing because gamma does not get the report

This comment was marked as abuse.

Sign in to view

typo and actually have Gamma comment check use gamma, not alpha

7dfb6ee

This comment was marked as abuse.

Sign in to view

RocketDerp and others added 2 commits July 19, 2023 15:35

prepare-drone-federation-test.sh has some more echo output and note a…

0384675

…bout the LEMMY_DATABASE_URL format (LemmyNet#3651)

Nutomic reviewed Jul 19, 2023

View reviewed changes

Nutomic added 2 commits July 20, 2023 11:58

Merge branch 'main' into RocketDerp-federation_tests_replication_round1

9769721

Rewrite activity lists to fix delete federation (fixes LemmyNet#3625)

71d5ae7

Nutomic reviewed Jul 20, 2023

View reviewed changes

RocketDerp added 2 commits July 20, 2023 06:09

Revert "typo and actually have Gamma comment check use gamma, not alpha"

5cd2852

This reverts commit 7dfb6ee.

Revert "Far more robust "Report a comment" test. Many comments about …

a72590b

…situation, this is currently failing because gamma does not get the report" This reverts commit 7bd3b20.

RocketDerp force-pushed the federation_tests_replication_round1 branch from 8a2bab8 to a72590b Compare July 20, 2023 13:10

RocketDerp closed this Jul 20, 2023

RocketDerp reopened this Jul 20, 2023

This comment was marked as abuse.

Sign in to view

Merge branch 'LemmyNet:main' into federation_tests_replication_round1

6c74f9f

This comment was marked as abuse.

Sign in to view

RocketDerp closed this Jul 20, 2023

RocketDerp reopened this Jul 20, 2023

Merge branch 'LemmyNet:main' into federation_tests_replication_round1

fb32d19

RocketDerp added 2 commits July 20, 2023 08:45

prettier TypeScript

24f3d02

revised comments, as ResolveObject isn't using routine replication

bdb7381

This comment was marked as abuse.

Sign in to view

fmt

46b7e6a

This comment was marked as abuse.

Sign in to view

dessalines and others added 3 commits July 26, 2023 17:38

Merge branch 'main' into federation_tests_replication_round1

a3658a9

fix api tests

9cb9ca3

remove comment

4e741fb

Nutomic approved these changes Jul 27, 2023

View reviewed changes

Nutomic enabled auto-merge (squash) July 27, 2023 09:53

Nutomic merged commit 21a87eb into LemmyNet:main Jul 27, 2023
1 check passed

This comment was marked as abuse.

Sign in to view

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Federation tests replication round1 - demonstrate absent replication of comment deletes #3657

Federation tests replication round1 - demonstrate absent replication of comment deletes #3657

RocketDerp commented Jul 18, 2023 •

edited

This comment was marked as abuse.

This comment was marked as abuse.

Nutomic Jul 19, 2023 •

edited

This comment was marked as abuse.

Nutomic Jul 19, 2023

This comment was marked as abuse.

Nutomic Jul 19, 2023

Nutomic Jul 19, 2023

This comment was marked as abuse.

Nutomic commented Jul 20, 2023 •

edited

Nutomic Jul 20, 2023

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

Nutomic commented Jul 21, 2023

This comment was marked as abuse.

Nutomic commented Jul 27, 2023

This comment was marked as abuse.

Federation tests replication round1 - demonstrate absent replication of comment deletes #3657

Federation tests replication round1 - demonstrate absent replication of comment deletes #3657

Conversation

RocketDerp commented Jul 18, 2023 • edited

This comment was marked as abuse.

This comment was marked as abuse.

Nutomic Jul 19, 2023 • edited

Choose a reason for hiding this comment

This comment was marked as abuse.

Nutomic Jul 19, 2023

Choose a reason for hiding this comment

This comment was marked as abuse.

Nutomic Jul 19, 2023

Choose a reason for hiding this comment

Nutomic Jul 19, 2023

Choose a reason for hiding this comment

This comment was marked as abuse.

Nutomic commented Jul 20, 2023 • edited

Nutomic Jul 20, 2023

Choose a reason for hiding this comment

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

Nutomic commented Jul 21, 2023

This comment was marked as abuse.

Nutomic commented Jul 27, 2023

This comment was marked as abuse.

RocketDerp commented Jul 18, 2023 •

edited

Nutomic Jul 19, 2023 •

edited

Nutomic commented Jul 20, 2023 •

edited