Firehose consumers do not receive root block in case of too big seq commit

[MarshalX]

Describe the bug

Firehose consumers do not receive root block in case of too big seq commit.

To Reproduce

I wrote a corresponding unit test in the linked PR that reproduces the issue

Actual behavior

car.blocks are always empty

Expected behavior

car.blocks must be always one (the root one)

Additional context

Line no. 29 is so suspicious:

atproto/packages/pds/src/sequencer/events.ts

Lines 25 to 31 in 81ae1b1

	// max 200 ops or 1MB of data
	if (writes.length > 200 || commitData.newBlocks.byteSize > 1000000) {
	tooBig = true
	const justRoot = new BlockMap()
	justRoot.add(commitData.newBlocks.get(commitData.cid))
	carSlice = await blocksToCarFile(commitData.cid, justRoot)
	} else {

Here is why:

• .add of BlockMap returns a promise and no one waits for resolve (forgotten await)
• input arg of .add is typed asLexValue which goes through dataToCborBlock(lexToIpld(value)). That does not make sense since we just move the block from one BlockMap to another one. newBlocks.get returns bytes already.

P.S. I dug into this because I got reports from my lovely Python devs that commit.blocks received from the firehose message frame could be missed. But it is marked as a required field in the lexicon. From our observation, it is missed always in combination with tooBig=true. I will continue my journey, but if you have any ideas I would appreciate it

[MarshalX]

Hi @dholms! I am pinging you here since you are the last one in the git blame for this file :)

[MarshalX]

bump :(

[bnewbold]

The intention going forward is that the commit block should be included as the root block. It is also possible it isn't getting passed through the relay? We'll need to investigate.

[dholms]

Hey @MarshalX thanks for flagging this!

I put up a fix for the PDS codebase (#2923). But we seldom if ever output tooBig events on the PDS. I'll flag this in the relay codebase as well which I think is likely the culprit of these missing commits

[dholms]

Oh rip I didn't see your PR on this 😅

Let me take a closer look, I might prefer yours