Publish 'raw' package for use in other apps + CLI

[0xdevalias]

It would be really cool if the raw functionality of this parser was able to be used as a package in other (non-web) apps, and/or if there was a CLI version of it.

[alvarlagerlof]

What would you expect to get as output from it? The parsing part of the codebase is modified React source. In React it has side effects, but in the code here it's stopping early at a few points so that it can return data instead.

[0xdevalias]

I was exploring how to parse the self.__next_f aspects of a Next.js v13+ app; and hacked together a really basic workaround to extract the bits I was interested in at that time; but when I saw this lib, I was kind of expecting/hoping that I might be able to 'more officially' delegate the parsing to it to be done 'properly':

• https://gist.github.com/0xdevalias/ac465fb2f7e6fded183c2a4273d21e61#react-server-components-nextjs-v13-and-webpack-notes-on-streaming-wire-format-__next_f-etc

  • const parseJSONFromEntry = entry => {
      const jsonPart = entry.substring(entry.indexOf('[') + 1, entry.lastIndexOf(']'));
      try {
        return JSON.parse(`[${jsonPart}]`);
      } catch (e) {
        console.error("Failed to parse JSON for entry: ", entry);
        return [];  // Return an empty array or null as per error handling choice
      }
    };
    
    // ..snip..
    
    // Get imports/etc (v3)
    const moduleDependencies = 
      self.__next_f
        .map(f => f?.[1])
        .filter(f => f?.includes('static/'))
        .flatMap(f => f.split('\n'))
        .map(parseJSONFromEntry)
        .filter(f => Array.isArray(f) ? f.length > 0 : !!f)
        .map(f => {
          if (!Array.isArray(f?.[1])) { return f } else {
            // Convert alternating key/value array to an object
            const keyValueArray = f[1];
            const keyValuePairs = [];
            for (let i = 0; i < keyValueArray.length; i += 2) {
              keyValuePairs.push([keyValueArray[i], keyValueArray[i + 1]]);
            }
            f[1] = Object.fromEntries(keyValuePairs);
            return f;
          }
        })
        .filter(f => Array.isArray(f) && f.length === 3 && typeof f?.[1] === 'object')
        .reduce((acc, [moduleId, dependencies, _]) => {
          acc[moduleId] = dependencies;
          return acc;
        }, {});
    
    // ..snip..

I guess I haven't thought too deeply as to the 'best' output format.. but I guess the most basic version that would be useful would be the equivalent of what is output in the 'Raw data' section of the site currently; where I could pass in 1 or more lines of 'raw RSC wire format', and get back the parsed versions of it.

[alvarlagerlof]

I think I could expose createFlightResponse from @rsc-parser/core. I'll make a PR.

[alvarlagerlof]

I think that this is the best reference for how to use it:

rsc-parser/packages/core/src/components/ViewerPayload.tsx

Line 51 in 0ceab13

const messages = [

[0xdevalias]

I think I could expose createFlightResponse from @rsc-parser/core. I'll make a PR.

@alvarlagerlof Awesome, thanks for that!

---

I think that this is the best reference for how to use it

@alvarlagerlof Skimming through the code there, it looks like it:

• sets up an array of messages to pass to createFlightResponse (which since your PR, is exposed from @rsc-parser/core as unstable_createFlightResponse

  • rsc-parser/packages/core/src/components/ViewerPayload.tsx

    Lines 51 to 66 in 0ceab13

    	const messages = [
    	{
    	type: "RSC_CHUNK",
    	tabId: 0,
    	data: {
    	fetchUrl: "https://example.com",
    	fetchMethod: "GET",
    	fetchStartTime: 0,
    	chunkStartTime: 0,
    	chunkEndTime: 0,
    	chunkValue: Array.from(new TextEncoder().encode(payload)),
    	},
    	} satisfies RscChunkMessage,
    	];
    	const flightResponse = createFlightResponse(messages);

• for each of the messages, createFlightResponse calls processBinaryChunk

  • rsc-parser/packages/core/src/createFlightResponse.ts

    Lines 9 to 34 in 347b3a4

    	export function createFlightResponse(
    	messages: RscChunkMessage[],
    	): FlightResponse {
    	// @ts-expect-error TODO: fix this
    	const response: FlightResponse = {
    	_buffer: [],
    	_rowID: 0,
    	_rowTag: 0,
    	_rowLength: 0,
    	_rowState: 0,
    	_currentStartTime: 0,
    	_currentEndTime: 0,
    	_stringDecoder: createStringDecoder(),
    	_chunks: [] as FlightResponse["_chunks"],
    	};
    	response._fromJSON = createFromJSONCallback(response);
    	for (let i = 0; i < messages.length; i++) {
    	response._currentStartTime = messages[i].data.chunkStartTime;
    	response._currentEndTime = messages[i].data.chunkEndTime;
    	processBinaryChunk(response, Uint8Array.from(messages[i].data.chunkValue));
    	}
    	return response;
    	}

• processBinaryChunk seems to do the deeper parsing of the actual RSC data (and calls into other functions like processFullRow/etc to do so)

Will need to have a bit of a play to understand how best to use this alongside my 'extract from the script tags' code; but thanks for exposing it! :)