chore: message.nim - set max message size to 150KiB according to spec #2298

Ivansete-status · 2023-12-14T15:07:27Z

Description

Change to align the actual message limit with the one written in the spec: https://rfc.vac.dev/spec/64/#message-size

Notice that the spect states 150KB but we are setting the maximum to 150KiB because that sounds more aligned with the actual default value defined within nim-libp2p (1024 * 1024)

Issue

closes #2297

github-actions · 2023-12-14T15:12:01Z

You can find the image built from this PR at

quay.io/wakuorg/nwaku-pr:2298

Built from b966524

fryorcraken · 2023-12-14T23:40:04Z

This needs to be configurable so that the Status fleet can retain the 1MB message size for now (note they already have plans to reduce their message size).

chaitanyaprem · 2023-12-15T09:00:22Z

~~Shouldn't this sizeCheck also be done at lightpush and respond with error in case message size crosses this limit?~~
Realized since lightpush message would anyways be sent via relay and hence the check would hit and fail.

github-actions · 2023-12-15T15:34:47Z

This PR may contain changes to configuration options of one of the apps.

If you are introducing a breaking change (i.e. the set of options in latest release would no longer be applicable) make sure the original option is preserved with a deprecation note for 2 following releases before it is actually removed.

Please also make sure the label release-notes is added to make sure any changes to the user interface are properly announced in changelog and release notes.

Ivansete-status · 2023-12-15T16:51:32Z

This needs to be configurable so that the Status fleet can retain the 1MB message size for now (note they already have plans to reduce their message size).

Hey @fryorcraken ! I've added the configurable parameter max-num-bytes-msg-size:

    maxMessageSize* {.
      desc: "Maximum message size in bytes. e.j. to set a 1 MiB, set this to 1048576 (1024*1024)."
      defaultValue: MaxWakuMessageSize
      name: "max-num-bytes-msg-size" }: int

gabrielmer

LGTM, thanks so much!

alrevuelta

lgtm. Note that some non TWN deployments would need to be updated.

fryorcraken · 2023-12-19T11:21:33Z

This needs to be configurable so that the Status fleet can retain the 1MB message size for now (note they already have plans to reduce their message size).

Hey @fryorcraken ! I've added the configurable parameter max-num-bytes-msg-size:
    maxMessageSize* {.
      desc: "Maximum message size in bytes. e.j. to set a 1 MiB, set this to 1048576 (1024*1024)."
      defaultValue: MaxWakuMessageSize
      name: "max-num-bytes-msg-size" }: int

Thanks for that. How do we ensure that we don't forget to change the config when upgrading Status fleet?

nitpick: could it accept 1MiB as input?

Ivansete-status · 2023-12-19T11:40:59Z

Thanks for that. How do we ensure that we don't forget to change the config when upgrading Status fleet?

Good point! We've created this issue to tackle that: #2305

nitpick: could it accept 1MiB as input?

I'll check it asap 👌

Using KiB instead of KB because that seems more aligned with the actual default defined in nim-libp2p (1024 * 1024) Spec details: https://rfc.vac.dev/spec/64/#message-size

…nits

… adjustment The "Fails to push message with large meta" test used 10 ** 6 when `nwaku` node had MaxWakuMessageSize == 1MiB ( 1*2^20 .) `nwaku` establishes the max lightpush msg size as `const MaxRpcSize* = MaxWakuMessageSize + 64 * 1024` see: https://github.com/waku-org/nwaku/blob/07beea02095035f4f4c234ec2dec1f365e6955b8/waku/waku_lightpush/rpc_codec.nim#L15 In the PR waku-org/nwaku#2298 we reduced the MaxWakuMessageSize from 1MiB to 150KiB. Therefore, the 105024 number comes from substracting ( 1*2^20 - 150*2^10 ) to the original 10^6 that this test had when MaxWakuMessageSize == 1*2^20

AlejandroCabeza

LGTM!

AlejandroCabeza · 2023-12-25T15:52:31Z

tests/waku_filter_v2/test_waku_client.nim

-          msg6 = fakeWakuMessage(contentTopic=contentTopic, payload=getByteSequence(3*1024*1024 + 1023*1024 + 970)) # 4MiB - 54B -> Out of Max Size
+          msg4 = fakeWakuMessage(contentTopic=contentTopic, payload=getByteSequence(MaxRpcSize - 1024)) # Max Size (Inclusive Limit)
+          msg5 = fakeWakuMessage(contentTopic=contentTopic, payload=getByteSequence(MaxRpcSize)) # Max Size (Exclusive Limit)
+          msg6 = fakeWakuMessage(contentTopic=contentTopic, payload=getByteSequence(MaxRpcSize)) # Out of Max Size


This one is duplicated

AlejandroCabeza · 2023-12-25T15:55:08Z

waku/waku_core/message/default_values.nim

+  ## https://rfc.vac.dev/spec/64/#message-size
+  MaxWakuMessageSize* = 150 * 1024 # Remember that 1 MiB is the PubSub default
+
+  DefaultMaxWakuMessageSizeStr* = "150KiB"


I wouldn't have two sources of truth (MaxWakuMessageSize and DefaultMaxWakuMessageSizeStr). How about we use the str one as source of truth, and define the other in terms of it by calling parseMsgSize(DefaultMaxWakuMessageSizeStr)?

Ivansete-status · 2023-12-29T10:51:12Z

I can't quite understand why the js-waku test is failing.
I run it locally and seems to work.
command: WAKUNODE_IMAGE=quay.io/wakuorg/nwaku-pr:2298 DEBUG=waku* npm run test:node
results:

NagyZoltanPeter

LGTM apart two little comment you may consider. Thank you.

NagyZoltanPeter · 2023-12-29T15:59:39Z

waku/common/utils/parse_size_units.nim

+  ## Parses size strings such as "1.2 KiB" or "3Kb" and returns the equivalent number of bytes
+  ## if the parse task goes well. If not, it returns an error describing the problem.
+
+  const RegexDef = """\s*(\d+([\,\.]\d*)?)\s*([Kk]{0,1}[i]?[Bb]{1})"""


I'm not pretty sure if it would not be suitable to apply a more restrictive pattern?
Like is someone mistakenly use it "1o24 KiB" it still accepts but results in 241024. Or "l50kb" is translated to 501000. I would clearly drop trailing white space or put it under a group that should be empty at the end.
Also I may change the middle whitespace to accept zero or only 1 of it.

NagyZoltanPeter · 2023-12-29T16:00:46Z

waku/waku_core/message/default_values.nim

+
+const
+  ## https://rfc.vac.dev/spec/64/#message-size
+  DefaultMaxWakuMessageSizeStr* = "150KiB" # Remember that 1 MiB is the PubSub default


Suggested change

DefaultMaxWakuMessageSizeStr* = "150KiB" # Remember that 1 MiB is the PubSub default

DefaultMaxWakuMessageSizeStr* = "150KiB" # Remember that 1024 KiB is the PubSub default

As MiB is not supported we might not refer it anywhere.

fryorcraken · 2024-01-01T23:43:49Z

I can't quite understand why the js-waku test is failing. I run it locally and seems to work. command: WAKUNODE_IMAGE=quay.io/wakuorg/nwaku-pr:2298 DEBUG=waku* npm run test:node results:

We have some test with 1MB messages. Pretty sure @fbarbu15 or the @waku-org/js-waku-developers is already on it.

fbarbu15 · 2024-01-03T08:47:06Z

WAKUNODE_IMAGE=quay.io/wakuorg/nwaku-pr:2298 DEBUG=waku* npm run test:node

In

I can't quite understand why the js-waku test is failing. I run it locally and seems to work. command: WAKUNODE_IMAGE=quay.io/wakuorg/nwaku-pr:2298 DEBUG=waku* npm run test:node results:

We have some test with 1MB messages. Pretty sure @fbarbu15 or the @waku-org/js-waku-developers is already on it.

Indeed as @fryorcraken suggested, some 1MB tests are failing with quay.io/wakuorg/nwaku-pr:2298 but on interop test side (will update those).
On js-waku side, we have 1MB tests but they do not fail probably because js-waku node is doing the sending. The job was failing because of an unrelated test that is passing now on re-run and the job is green cc @Ivansete-status

Ivansete-status self-assigned this Dec 14, 2023

Ivansete-status marked this pull request as ready for review December 14, 2023 15:07

Ivansete-status requested review from richard-ramos, AlejandroCabeza, SionoiS, alrevuelta, jm-clius, gabrielmer, fryorcraken, NagyZoltanPeter, vpavlin and chaitanyaprem December 14, 2023 15:08

chaitanyaprem mentioned this pull request Dec 15, 2023

feat: relay msg size waku-org/go-waku#963

Merged

gabrielmer approved these changes Dec 18, 2023

View reviewed changes

alrevuelta approved these changes Dec 18, 2023

View reviewed changes

Ivansete-status mentioned this pull request Dec 18, 2023

chore: update the status.* fleets so that they keep using 1MiB as maximum relay message size #2305

Closed

Ivansete-status changed the title ~~chore: message.nim - set max message size to 150KB according to spec~~ chore: message.nim - set max message size to 150KiB according to spec Dec 20, 2023

Ivansete-status added 6 commits December 21, 2023 22:47

message.nim: set max message size to 150KiB according to spec

fbcfbfb

Using KiB instead of KB because that seems more aligned with the actual default defined in nim-libp2p (1024 * 1024) Spec details: https://rfc.vac.dev/spec/64/#message-size

test_protocol.nim: align test to current WakuMessage limit

65379ab

test_waku_client.nim: adapt test to MaxWakuMessageSize change

4b1ad14

make maxMessageSize configurable for wakunode2

189bbec

wakunode2 app now accepts max-num-bytes-msg-size with KiB, KB, or B u…

4ed576c

…nits

testlib/wakunode.nim: set maxMessageSize: "1024 KiB"

79ae7ba

Ivansete-status force-pushed the set-max-size-to-150kb-as-spec branch from 8498a56 to 79ae7ba Compare December 21, 2023 21:50

Ivansete-status mentioned this pull request Dec 22, 2023

test: light-push/index.node.spec.ts: adjust metadata size to nwaku max size adjustment waku-org/js-waku#1769

Merged

Ivansete-status mentioned this pull request Dec 23, 2023

[Deliverable] DOS protection for req-res protocols and metrics waku-org/pm#66

Open

3 tasks

AlejandroCabeza approved these changes Dec 25, 2023

View reviewed changes

Ivansete-status added 4 commits December 28, 2023 12:54

test_waku_client.nim: remove duplicate check in "Valid Payload Sizes"

950fb38

set DefaultMaxWakuMessageSizeStr as the only source of truth

630b996

tests/common/test_parse_size.nim: cast to uint64

81d8ef2

external_config.nim: rename max-num-bytes-msg-size -> max-msg-size

a4a4936

NagyZoltanPeter approved these changes Dec 29, 2023

View reviewed changes

fryorcraken approved these changes Jan 1, 2024

View reviewed changes

vpavlin approved these changes Jan 3, 2024

View reviewed changes

Ivansete-status merged commit ed09074 into master Jan 3, 2024
10 checks passed

Ivansete-status deleted the set-max-size-to-150kb-as-spec branch January 3, 2024 12:11

fbarbu15 mentioned this pull request Jan 3, 2024

fix for 150 kb max messages adjustments waku-org/waku-interop-tests#10

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: message.nim - set max message size to 150KiB according to spec #2298

chore: message.nim - set max message size to 150KiB according to spec #2298

Ivansete-status commented Dec 14, 2023 •

edited

github-actions bot commented Dec 14, 2023 •

edited

fryorcraken commented Dec 14, 2023

chaitanyaprem commented Dec 15, 2023 •

edited

github-actions bot commented Dec 15, 2023

Ivansete-status commented Dec 15, 2023

gabrielmer left a comment

alrevuelta left a comment

fryorcraken commented Dec 19, 2023

Ivansete-status commented Dec 19, 2023

AlejandroCabeza left a comment

AlejandroCabeza Dec 25, 2023

AlejandroCabeza Dec 25, 2023

Ivansete-status commented Dec 29, 2023

NagyZoltanPeter left a comment

NagyZoltanPeter Dec 29, 2023

NagyZoltanPeter Dec 29, 2023

fryorcraken commented Jan 1, 2024

fbarbu15 commented Jan 3, 2024 •

edited

	DefaultMaxWakuMessageSizeStr* = "150KiB" # Remember that 1 MiB is the PubSub default
	DefaultMaxWakuMessageSizeStr* = "150KiB" # Remember that 1024 KiB is the PubSub default

chore: message.nim - set max message size to 150KiB according to spec #2298

chore: message.nim - set max message size to 150KiB according to spec #2298

Conversation

Ivansete-status commented Dec 14, 2023 • edited

Description

Issue

github-actions bot commented Dec 14, 2023 • edited

fryorcraken commented Dec 14, 2023

chaitanyaprem commented Dec 15, 2023 • edited

github-actions bot commented Dec 15, 2023

Ivansete-status commented Dec 15, 2023

gabrielmer left a comment

Choose a reason for hiding this comment

alrevuelta left a comment

Choose a reason for hiding this comment

fryorcraken commented Dec 19, 2023

Ivansete-status commented Dec 19, 2023

AlejandroCabeza left a comment

Choose a reason for hiding this comment

AlejandroCabeza Dec 25, 2023

Choose a reason for hiding this comment

AlejandroCabeza Dec 25, 2023

Choose a reason for hiding this comment

Ivansete-status commented Dec 29, 2023

NagyZoltanPeter left a comment

Choose a reason for hiding this comment

NagyZoltanPeter Dec 29, 2023

Choose a reason for hiding this comment

NagyZoltanPeter Dec 29, 2023

Choose a reason for hiding this comment

fryorcraken commented Jan 1, 2024

fbarbu15 commented Jan 3, 2024 • edited

Ivansete-status commented Dec 14, 2023 •

edited

github-actions bot commented Dec 14, 2023 •

edited

chaitanyaprem commented Dec 15, 2023 •

edited

fbarbu15 commented Jan 3, 2024 •

edited