Misskey v13 failed to communicate with some of the mastodon sites and old misskey sites via relay

[CHN-beta]

💡 Summary

After upgrade to v13, my Misskey instance failed to communicate with some of the other instances via relay. That is, for example, with misskey v13, most of the notes from o3o.ca no longer be shown in the global timeline, and the requests to o3o.ca showed a significant drop (red box in the picture).

Meanwhile, old misskey instances (v12) could not communicate to v13 in a similar manner.

This behavior could be reproduced in a newly set instance.

🥰 Expected Behavior

All notes from mastodon and misskey v12 instances connected to the same relay should be shown in global timeline.

🤬 Actual Behavior

On a new set misskey v13 instances, a lot of notes from misskey v13 instances were shown in global timeline. No note from mastodon was shown.

On a new set misskey v12 instances, only notes from mastodon were shown in global timeline. Notes from misskey v12 instances might could also be shown in global timeline, but today active v12 instances are not common.

📝 Steps to Reproduce

1. Setup two new instance, use v12.119.2 and v13.2.5 respectively. The two instances use similar configs:

url: https://misskeytestv12.chn.moe/
port: 3004
db:
  host: 127.0.0.1
  port: 5432
  db: misskeytestv12
  user: xxx
  pass: xxxxxxxxxxxxxxx
  disableCache: true
  extra:
    statement_timeout: 600000
redis:
  host: localhost
  port: 6379
id: 'aid'
proxyRemoteFiles: true
signToActivityPubGet: true

2. Do not follow or search any one, Just add a relay: https://relay.nya.one/inbox
3. Wait for several minutes. In the global timeline of misskey v12 instance, only a lot of notes from mastodon was shown. In the global timeline of misskey v13 instance, only a lot of notes from misskey v13 was shown.

A screenshot from v12 instance:

A screenshot from v13 instance:

📌 Environment

Misskey version: 13.2.5, running with docker
Your OS: Gentoo
Your browser: firefox

[CHN-beta]

The log files.

log.tar.gz

[rakino]

Chart for Misskey.cf notes

[Candinya]

Same question, still trying to figure out differences between:

1. directly delivered non-misskey notes
2. relay delivered non-misskey notes
3. relay delivered misskey notes

Currently only 2. relay delivered non-misskey notes has this weird issue (the others can show up in global timeline site (global timeline and user's page) normally)

They all have been observed from inbox queue, so it might be a queue processor issue (I guess).

Only for reference, not 100% sure.

[CHN-beta]

Same question, still trying to figure out differences between:

1. directly delivered non-misskey notes

2. relay delivered non-misskey notes

3. relay delivered misskey notes

Currently only 2. relay delivered non-misskey notes has this weird issue (the others can show up in global timeline site (global timeline and user's page) normally)

They all have been observed from inbox queue, so it might be a queue processor issue (I guess).

Only for reference, not 100% sure.

According to my test, relay delivered notes from misskey v13 also have some issue: They did now show up in global timeline in misskey v12.

[CHN-beta]

I have confirmed that the problem behavior could be reproduced at v12.120.0-alpha7 (commit b19ef59). That is, v12.120.0-alpha7 behaves like v13.2.5, it could only receive notes by relay from misskey v13 but not from mastodon.
I have also tried two version before v12.120.0-alpha7 (v12.120.0-alpha4 and v12.120.0-alpha1), sadly both of them failed to start.

[HolgerHuo]

FYI, notes from BirdSiteLive are not delivered as well.

Those ActivityPub create activities were delivered successfully, from both BirdSiteLive's and Activity-Relay's perspectives (no error logs), and also Misskey's perspective (as it responed with status code 202):

Nginx log for BirdSiteLive activity: "POST /inbox HTTP/2.0" 202 0 "-" "-"
Nginx log for Activity-Relay activity: "POST /inbox HTTP/2.0" 202 0 "-" "Relay Service (golang net/http; Activity-Relay ; relay.example.com)"

[AyumuNekozuki]

自身の環境でも同様の現象が発生し、色々調べたところ、以下のログが発生していることを確認しました。
※ NODE_ENV=production を付与せずにサーバーを起動すると以下のログが確認できます。

completed(skip: LD-Signatureの検証に失敗しました) id=23811 attempts=1/8 age=563ms

暫定的な対処として、以下のコードをコメントアウトし、再ビルド・起動すると問題なく、リレーからの投稿が流れるようになります。ただ、これは "正しい" 対処法ではない気がします。

misskey/packages/backend/src/queue/processors/InboxProcessorService.ts

Lines 147 to 152 in 44b9539

	// LD-Signature検証
	const ldSignature = this.ldSignatureService.use();
	const verified = await ldSignature.verifyRsaSignature2017(activity, authUser.key.keyPem).catch(() => false);
	if (!verified) {
	return 'skip: LD-Signatureの検証に失敗しました';
	}

[Candinya]

Seems like the normalize function has changed

Before:

misskey/packages/backend/src/remote/activitypub/misc/ld-signature.ts

Lines 79 to 84 in 91f4ec3

	public async normalize(data: any) {
	const customLoader = this.getLoader();
	return await jsonld.normalize(data, {
	documentLoader: customLoader,
	});
	}

After:

misskey/packages/backend/src/core/activitypub/LdSignatureService.ts

Lines 85 to 88 in 5b38f76

	public async normalize(data: any) {
	const customLoader = this.getLoader();
	return 42;
	}

[Candinya]

Fix confirmed, will open a PR for this.
Note: this will be a breaking change, any instance running v13 should update immediately to ensure the correct signature