Backfill remote asset listing when needed #1531

brandonpayton · 2024-06-21T02:57:47Z

Motivation for the change, related issues

In order to reintroduce the request routing improvements from #1490, we need to be sure that existing browser storage can backfill the wordpress-remote-asset-paths file when needed. That is the purpose of this PR.

Implementation details

During boot, we check whether the /wordpress/wordpress-remote-asset-paths exists. If it does not exist, we check a static file path that should be common across WP versions, wp-admin/css/common.css. If the common static file also does not exist, we guess we are dealing with a minified WP build cached in browser storage without a remote asset listing file, and we download it based on detected WordPress version (as long as the detected version is a supported WP version).

Testing Instructions (or ideally a Blueprint)

So far, I have tested this manually with steps like the following:

Install the Chrome extension OPFS Explorer
Switch Playground to browser storage and note the current WP release X in the query string
Open dev tools
Use OPFS explorer to confirm /wordpress/wordpress-remote-asset-paths exists and then delete the file
Switch to the dev tools Network tab
Reload the page
Filter the requests and confirm there was a request for the current WP release X like /wp-<X>/wordpress-remote-asset-paths
Clear the Network request log
Use OPFS explorer to confirm /wordpress/wordpress-remote-asset-paths exists again. Then delete the file.
Manually adjust the query string to point to request a different WP release.
Reload the page
Return to the dev tools Network tab, filter the requests, and confirm there was a request for the current WP release X like /wp-<X>/wordpress-remote-asset-paths. Regardless of the WP version in the query string, we should continue requesting the paths listing for version X because that is what is in browser storage.
Review the console log and confirm there is a warning like "Loaded WordPress version (X) differs from requested version (Y)" and confirm that Y matches the version requested via the query string.
Use OPFS explorer to confirm /wordpress/wordpress-remote-asset-paths exists again.

bgrgicak · 2024-06-21T09:30:28Z

@brandonpayton could we add some tests for this PR?

brandonpayton · 2024-06-21T13:10:02Z

@brandonpayton could we add some tests for this PR?

Yep!

The boot process doesn't need to know about web-related compromises. That can be handled by the process that invokes bootWordPress().

brandonpayton · 2024-06-24T04:46:17Z

packages/playground/wordpress-builds/src/index.ts

+	wpVersion: string
+): string | undefined {
+	return wpVersion in SupportedWordPressVersions
+		? `wp-${wpVersion}`


The purpose of this is to explicitly export the knowledge of where static files are located and to avoid duplicating that knowledge elsewhere.

brandonpayton · 2024-06-24T04:54:59Z

@bgrgicak I reworked this to remove the static asset list retrieval from boot and move it to worker-thread which is AFAIK is a better, web-specific place for it.

@brandonpayton could we add some tests for this PR?

Yep!

I'm not sure yet but may have to withdraw the "Yep!". It may be possible to check this with e2e tests, and I plan to look at that in the morning.

brandonpayton · 2024-06-24T17:46:12Z

I updated this PR with manual testing instructions and am considering whether we might add e2e tests for this.

brandonpayton · 2024-06-24T17:49:18Z

The next step after this PR is to re-add the better request routing based on the remote asset path lookup. Should be straightforward now that can count on the listing file existing when needed.

brandonpayton · 2024-06-24T17:59:21Z

I had a TODO to consider a joinPaths() function that works better joining parts of URLs that include a schema like "https://" because joinPaths() isn't meant for that and condenses the slashes in https:// to https:/.

But another function is unnecessary. Just treat the path and base separate using the new URL(path, base) constructor like:

new URL(
    joinPaths(...),
    schemeHostAndBasePath
)

adamziel · 2024-06-24T20:06:54Z

If the common static file also does not exist, we guess we are dealing with a minified WP build cached in browser storage without a remote asset listing file

Something feels off about this check. Imagine we're loading a minified build and stream-backfilling all the static assets. At one point, that CSS file will be created which means it will be detected on the next load. That doesn't mean all the files are there. Perhaps checking for the presence of explicitly created wordpress-remote-asset-paths would suffice?

adamziel · 2024-06-24T20:08:41Z

packages/playground/remote/src/lib/worker-thread.ts

+
+	if (
+		wpStaticAssetsDir !== undefined &&
+		!primaryPhp.fileExists(remoteAssetListPath) &&


Shouldn't this check if the remoteAssetListPath exists (vs doesn't exist)?

Suggested change

!primaryPhp.fileExists(remoteAssetListPath) &&

primaryPhp.fileExists(remoteAssetListPath) &&

Oh, I see, you're checking if that file is missing from the WordPress build and if we need to fetch it. Hm. I'm not sure I follow – I thought that list is only supposed to exist in the minified builds? Official WordPress releases don't ship one.

I think it is correct as-is because we want to retrieve the file on-demand if it does not already exist.

What would be a case where it doesn't exist in the minified zip bundle but does exist on the server?

Aha, that would be the case for builds that are already cached in the browser storage today.

Oh, I see, you're checking if that file is missing from the WordPress build and if we need to fetch it. Hm. I'm not sure I follow – I thought that list is only supposed to exist in the minified builds? Official WordPress releases don't ship one.

@adamziel That's the challenge here. We don't want to retrieve this listing unless it is needed. It normally just comes with minified WP builds, but once we start depending on the existence of the listing file (#1532) and encounter an older minified WP build in browser or device storage, we need to retrieve the listing so the minified build continues to work. That was also the motivation of checking for the single wp-admin/css/common.css file as a hint that we are potentially dealing with a minified build.

Do you have another idea about how we could or should approach this so we don't retrieve anything unnecessarily or add to the filesystem of full WP builds? Note this is the worker-thread in @wp-playground/remote, in case that influences your thinking.

cc @bgrgicak because I think the question of detecting full WP also affects #1532.

What is it with our timing here? 😄 I keep responding just as you respond in a way that makes my comment stale. 😂

Thanks for looking at this again today, @adamziel!

Aha, that would be the case for builds that are already cached in the browser storage today.

This is actually why I reverted the routing changes the second time... because of existing browser storage without the file.

brandonpayton · 2024-06-24T20:17:54Z

If the common static file also does not exist, we guess we are dealing with a minified WP build cached in browser storage without a remote asset listing file

Something feels off about this check. Imagine we're loading a minified build and stream-backfilling all the static assets. At one point, that CSS file will be created which means it will be detected on the next load. That doesn't mean all the files are there. Perhaps checking for the presence of explicitly created wordpress-remote-asset-paths would suffice?

It was intended as a sort of better-than-nothing check to avoid fetching the remote asset listing when we already have a full WP install or something like that. But what we could do is this:

If /wordpress/wordpress-remote-asset-paths does not exist, retrieve it.
When reading the file, filter out all remote asset paths that already exist and save what is left, even if nothing is left.

Pro: This would mean only retrieving the file once if needed.
Con: This would mean adding a non-core file to WP files in device storage.

🤔 Perhaps we could avoid this altogether when using device storage.

brandonpayton · 2024-06-24T20:25:21Z

Con: This would mean adding a non-core file to WP files in device storage.

🤔 Perhaps we could avoid this altogether when using device storage.

Nevermind. Today we are just loading Playground WP builds in device storage AFAICT.

To keep things simple, I'll reduce this to a simple check for listing existence, retrieve once, and write the contents without filtering.

brandonpayton · 2024-06-24T20:31:02Z

The questionable common-file check is now removed.

adamziel

Looking good, thank you @brandonpayton!

brandonpayton · 2024-06-25T01:37:57Z

@bgrgicak I took some time to try to add an e2e test to cover this dynamic retrieval as part of the e2e tests added in #1539, but I'm getting strange results where the listing file is asserted to exist but encounters a file-does-not exist error when trying to unlink. It would be good to cover more of this worker-thread logic with automated tests, but I'm going to move on for now.

Backfill remote asset listing when needed

0a4d992

brandonpayton added [Type] Enhancement New feature or request [Package][@php-wasm] Universal [Feature] Boot Flow labels Jun 21, 2024

brandonpayton requested a review from a team June 21, 2024 02:57

brandonpayton self-assigned this Jun 21, 2024

Keep WP assets dir knowledge with build package

91083fa

brandonpayton mentioned this pull request Jun 24, 2024

Download all WordPress assets on boot #1532

Merged

Move backfilling out of boot and into worker-thread

3e6d6ea

The boot process doesn't need to know about web-related compromises. That can be handled by the process that invokes bootWordPress().

brandonpayton force-pushed the remote-asset-list-on-demand branch from 585b9fd to 3e6d6ea Compare June 24, 2024 04:43

brandonpayton commented Jun 24, 2024

View reviewed changes

brandonpayton added 2 commits June 24, 2024 13:26

Address remaining TODOs

5705069

Tweak comment about common file

5106d9f

Fix unused import lint error

bd72f2f

brandonpayton mentioned this pull request Jun 24, 2024

Route requests more like a normal web server #1539

Merged

adamziel reviewed Jun 24, 2024

View reviewed changes

Remove questionable common-file check

017b0d3

adamziel approved these changes Jun 24, 2024

View reviewed changes

brandonpayton merged commit 4c8b417 into trunk Jun 25, 2024
5 checks passed

brandonpayton deleted the remote-asset-list-on-demand branch June 25, 2024 01:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Backfill remote asset listing when needed #1531

Backfill remote asset listing when needed #1531

brandonpayton commented Jun 21, 2024 •

edited

Loading

bgrgicak commented Jun 21, 2024

brandonpayton commented Jun 21, 2024

brandonpayton Jun 24, 2024 •

edited

Loading

adamziel Jun 24, 2024

brandonpayton commented Jun 24, 2024

brandonpayton commented Jun 24, 2024

brandonpayton commented Jun 24, 2024

brandonpayton commented Jun 24, 2024

adamziel commented Jun 24, 2024

adamziel Jun 24, 2024

adamziel Jun 24, 2024

brandonpayton Jun 24, 2024

adamziel Jun 24, 2024 •

edited

Loading

adamziel Jun 24, 2024

brandonpayton Jun 24, 2024

brandonpayton Jun 24, 2024 •

edited

Loading

brandonpayton Jun 24, 2024

brandonpayton commented Jun 24, 2024 •

edited

Loading

brandonpayton commented Jun 24, 2024

brandonpayton commented Jun 24, 2024

adamziel left a comment

brandonpayton commented Jun 25, 2024

	!primaryPhp.fileExists(remoteAssetListPath) &&
	primaryPhp.fileExists(remoteAssetListPath) &&

Backfill remote asset listing when needed #1531

Backfill remote asset listing when needed #1531

Conversation

brandonpayton commented Jun 21, 2024 • edited Loading

Motivation for the change, related issues

Implementation details

Testing Instructions (or ideally a Blueprint)

bgrgicak commented Jun 21, 2024

brandonpayton commented Jun 21, 2024

brandonpayton Jun 24, 2024 • edited Loading

Choose a reason for hiding this comment

adamziel Jun 24, 2024

Choose a reason for hiding this comment

brandonpayton commented Jun 24, 2024

brandonpayton commented Jun 24, 2024

brandonpayton commented Jun 24, 2024

brandonpayton commented Jun 24, 2024

adamziel commented Jun 24, 2024

adamziel Jun 24, 2024

Choose a reason for hiding this comment

adamziel Jun 24, 2024

Choose a reason for hiding this comment

brandonpayton Jun 24, 2024

Choose a reason for hiding this comment

adamziel Jun 24, 2024 • edited Loading

Choose a reason for hiding this comment

adamziel Jun 24, 2024

Choose a reason for hiding this comment

brandonpayton Jun 24, 2024

Choose a reason for hiding this comment

brandonpayton Jun 24, 2024 • edited Loading

Choose a reason for hiding this comment

brandonpayton Jun 24, 2024

Choose a reason for hiding this comment

brandonpayton commented Jun 24, 2024 • edited Loading

brandonpayton commented Jun 24, 2024

brandonpayton commented Jun 24, 2024

adamziel left a comment

Choose a reason for hiding this comment

brandonpayton commented Jun 25, 2024

brandonpayton commented Jun 21, 2024 •

edited

Loading

brandonpayton Jun 24, 2024 •

edited

Loading

adamziel Jun 24, 2024 •

edited

Loading

brandonpayton Jun 24, 2024 •

edited

Loading

brandonpayton commented Jun 24, 2024 •

edited

Loading