Read Webpack stats file as stream to support very large projects

[ethriel3695]

This builds off of the changes to the trim-stats-file.
We are now moving to change all instances of the stats file to use readStatsFile which will read stats files from a stream.
This will allow handling larger stats files and avoid hitting a file size limit.

[linear]

AP-2270 Add trim-stats-file changes to the preview-stats process

Now that we've tested that the stream read works correctly with trim-stats-file we can make changes to the preview-stats read process and get this working correctly.

[ethriel3695]

I added this check because I found that in the upload.test.ts test for traced files, it was erroring out because the file path does not return an existing file.

I attempted to mock the file path but because we mock fs-extra it overrides all my mock tests and cannot find the file.

The logic check ensures that we don't error out on a stats file with a directory that we cannot process.

However, once the trace or upload reaches the point where it's processing the stats file, we have already checked for a file path and that the file exists so we should be able to reasonably assume that the file exists and this is probably an indicator that the test needs to be refactored.

[ethriel3695]

What is strange is that when you include a combination of the sourceDir and statsPath the upload process builds a path that would most likely not exist in production.

sourceDir: '/static/',
fileInfo: { statsPath: '/static/preview-stats.json' },

This is the path: static/static/preview-stats.json
This is the existing test so I'm hesitant to change it but that file path does not seem like something that would ever exist in a real project.

So is the test incorrect, is the path join incorrect, or is this expected?
@ghengeveld @tmeasday

[tmeasday]

@ethriel3695 reading where statsPath comes from, it is relative to the sourceDir:

chromatic-cli/bin-src/tasks/upload.ts

Lines 77 to 117 in 0d1bd21

	// Get all paths in rootDir, starting at dirname.
	// We don't want the paths to include rootDir -- so if rootDir = storybook-static,
	// paths will be like iframe.html rather than storybook-static/iframe.html
	function getPathsInDir(ctx: Context, rootDir: string, dirname = '.'): PathSpec[] {
	try {
	return fs.readdirSync(join(rootDir, dirname)).flatMap((p: string) => {
	const pathname = join(dirname, p);
	const stats = fs.statSync(join(rootDir, pathname));
	return stats.isDirectory()
	? getPathsInDir(ctx, rootDir, pathname)
	: [{ pathname, contentLength: stats.size }];
	});
	} catch (e) {
	ctx.log.debug(e);
	throw new Error(invalid({ sourceDir: rootDir } as any, e).output);
	}
	}
	function getOutputDir(buildLog: string) {
	const outputString = 'Output directory: ';
	const outputIndex = buildLog.lastIndexOf(outputString);
	if (outputIndex === -1) return undefined;
	const remainingLog = buildLog.slice(outputIndex + outputString.length);
	const newlineIndex = remainingLog.indexOf('\n');
	const outputDir = newlineIndex === -1 ? remainingLog : remainingLog.slice(0, newlineIndex);
	return outputDir.trim();
	}
	function getFileInfo(ctx: Context, sourceDir: string) {
	const lengths = getPathsInDir(ctx, sourceDir).map((o) => ({ ...o, knownAs: slash(o.pathname) }));
	const total = lengths.map(({ contentLength }) => contentLength).reduce((a, b) => a + b, 0);
	const paths: string[] = [];
	let statsPath: string;
	// eslint-disable-next-line no-restricted-syntax
	for (const { knownAs } of lengths) {
	if (knownAs.endsWith('preview-stats.json')) statsPath = knownAs;
	else if (!knownAs.endsWith('manager-stats.json')) paths.push(knownAs);
	}
	return { lengths, paths, statsPath, total };
	}

So I think that simply the test is wrong here.

[yannbf]

This all looks good to me, thanks for the good work @ethriel3695 !

[thafryer]

I like!! Good work @ethriel3695 !!

[tmeasday]

Couple comments!

[tmeasday]

How was this file previously mocking the reading of JSON?

[yannbf]

It wasn't. Just resolved to undefined and did not break. The tests could be improved if there are use cases that require stats files to contain something though

[tmeasday]

🙌