[AssetMapper] Change default importmap "provider" to JsDelivr+esm #50408

weaverryan · 2023-05-24T02:50:12Z

Q	A
Branch?	6.3
Bug fix?	yes
New feature?	yes
Deprecations?	no
Tickets	None
License	MIT
Doc PR	Still TODO

We currently use the https://jspm.org/ API in importmap:require to find a CDN URL for an npm package - just like Rails. Unfortunately, this is NOT as robust as we had thought. For me, it's broken. 3 big issues:

A) Not Combined
Some packages are not packed/combined. Example: chart.js/auto imports other packages and results in 3 requests instead of 1. Not TERRIBLE... so here IS a terrible example: @popperjs/core (needed by bootstrap) results in nearly 50 requests ❗

B) Downloading Broken
For some packages, downloading simply doesn't work - rails/importmap-rails#65. @popperjs/core is another good example. Many of its imports have the form import"./utils/getOppositeVariationPlacement.js. If we download the main file, it looks locally for that utils/ file, which won't be there. @chart.js/auto is another example.

C) process.env.NODE_ENV included
Some packages (yes, again @popperjs/core is a great example!) contain process.env.NODE_ENV inside - rails/importmap-rails#65 (comment)

I believe that some package advertise an "esm" package... but just don't do a good job of creating it... or create it without the browser context in mind (or at least in a way that's inconvenient for downloading).

jsDelivr to the rescue

THANKFULLY, jsDelivr seems to have a fantastic API/hosting that is almost exactly what we want: https://www.jsdelivr.com/?docs=esm

They deliver fully "packaged" modules, where the only import is for peer dependencies - e.g. https://cdn.jsdelivr.net/npm/bootstrap@5.2.3/+esm

There IS still an issue when downloading. "Peer imports" are relative -e.g. import*as t from"/npm/@popperjs/core@2.11.7/+esm"; However, these imports follow a VERY strict pattern. So, when --download is passed, we parse these, download the peer dependency and update the import contents to @popperjs/core, which works with the importmap. It's not ideal that we need to do that, but it's straightforward and works great.

Sorry again for this late PR - I had assumed that jspm was robust because Rails is using it. It turns out it's robust... unless you hit a "bad" package, then it's terrible. And they're not that rare: on ux.symfony.com, I have hit several.

Thanks!

carsonbot · 2023-05-24T02:50:18Z

Hey!

Thanks for your PR. You are targeting branch "6.4" but it seems your PR description refers to branch "6.3".
Could you update the PR description or change target branch? This helps core maintainers a lot.

Cheers!

Carsonbot

weaverryan · 2023-05-24T03:16:00Z

The "high-deps" failure is because it's trying to pull in the latest commit from symfony/framework-bundle, but that is missing the commit from this PR which handles the wiring. But I'm not sure how to fix that.

stof · 2023-05-24T08:29:04Z

The failing high-deps is because your Pr does not target the highest version (as the 6.4 branch has been created). It is fine to ignore that failure as it will be resolved once your merged PR is merged up from 6.3 to 6.4.

src/Symfony/Component/AssetMapper/ImportMap/Providers/JsDelivrEsmImportMapProvider.php

src/Symfony/Component/AssetMapper/Tests/ImportMap/ImportMapManagerTest.php

dunglas

Awesome!!

src/Symfony/Component/AssetMapper/ImportMap/Providers/JsDelivrEsmImportMapProvider.php

src/Symfony/Component/AssetMapper/ImportMap/Providers/JspmImportMapProvider.php

src/Symfony/Component/AssetMapper/ImportMap/Providers/ResolvedImportMapPackage.php

nicolas-grekas

I pushed a 2nd commit that addresses @stof's and @dunglas' comments + does some CS changes but more importantly fixes wiring of resolvers, sends requests in parallel and rewrites the MappedAsset class to use readonly properties.

nicolas-grekas · 2023-05-24T15:24:09Z

src/Symfony/Bundle/FrameworkBundle/DependencyInjection/FrameworkExtension.php

+
+        $container
+            ->getDefinition('asset_mapper.importmap.resolver')
+            ->replaceArgument(0, $config['provider'])


not processing the value at compile-time is important to make this configurable by env var

nicolas-grekas · 2023-05-24T15:24:28Z

src/Symfony/Bundle/FrameworkBundle/DependencyInjection/FrameworkExtension.php

        ;

        $container
            ->getDefinition('asset_mapper.importmap.renderer')
            ->replaceArgument(2, $config['importmap_polyfill'] ?? ImportMapManager::POLYFILL_URL)
            ->replaceArgument(3, $config['importmap_script_attributes'])
        ;
+
+        $container->registerForAutoconfiguration(PackageResolverInterface::class)
+            ->addTag('asset_mapper.importmap.resolver');


this new tag allows defining custom package resolvers in case one wants to host their own

nicolas-grekas · 2023-05-24T15:28:24Z

src/Symfony/Component/AssetMapper/ImportMap/Resolver/JsDelivrEsmResolver.php

+    public function __construct(
+        HttpClientInterface $httpClient = null,
+        private readonly string $versionUrlPattern = self::URL_PATTERN_VERSION,
+        private readonly string $distUrlPattern = self::URL_PATTERN_DIST,


I made those URLs configurable to make it possible to reuse the code while self-hosting somewhere else.

nicolas-grekas · 2023-05-24T15:28:58Z

src/Symfony/Component/AssetMapper/ImportMap/Resolver/JsDelivrEsmResolver.php

+            }
+
+            $response = $this->httpClient->request('GET', sprintf($this->versionUrlPattern, $packageName, urlencode($constraint)));
+            $requiredPackages[] = [$options, $response, $packageName, $filePath, $options];


by stacking HTTP requests, we can fetch versions concurrently

nicolas-grekas · 2023-05-24T15:29:42Z

src/Symfony/Component/AssetMapper/ImportMap/Resolver/JsDelivrEsmResolver.php

+            }
+
+            $version = $response->toArray()['version'];
+            $requiredPackages[$i][1] = $this->httpClient->request('GET', sprintf($this->distUrlPattern, $packageName, $version, $filePath));


and we can also concurrently fetch package dists

nicolas-grekas · 2023-05-24T15:39:09Z

src/Symfony/Component/AssetMapper/ImportMap/Resolver/JspmResolver.php

+        $resolvedPackages = [];
+        foreach ($response->toArray()['map']['imports'] as $packageName => $url) {
+            $options = $packageRequiresByName[$packageName] ?? new PackageRequireOptions($packageName, null, $defaultOptions->download, $defaultOptions->preload);
+            $resolvedPackages[] = [$options, $url, $options->download ? $this->httpClient->request('GET', $url, ['base_uri' => $this->baseUri]) : null];


fetch contents from jspm concurrently also

nicolas-grekas · 2023-05-24T15:40:32Z

src/Symfony/Component/AssetMapper/MappedAsset.php

-    {
-    }
-
-    public function getLogicalPath(): string


replacing most getters+setters by readonly props (except for "dependencies" arrays, which need mutability)

… readonly properties in MappedAsset

weaverryan

Looks great - I checked the changes, I don't see any issues. It's more polished - you accomplished the immutability with MappedAsset, which I couldn't quite pull off. Thank you!

weaverryan · 2023-05-24T16:19:07Z

src/Symfony/Component/AssetMapper/ImportMap/Resolver/JsDelivrEsmResolver.php

+        }
+
+        return array_values($resolvedPackages);
+    }


nicolas-grekas · 2023-05-24T19:06:21Z

Thank you @weaverryan.

carsonbot added Status: Needs Review AssetMapper Bug Feature labels May 24, 2023

carsonbot added this to the 6.4 milestone May 24, 2023

weaverryan changed the base branch from 6.4 to 6.3 May 24, 2023 02:50

weaverryan modified the milestones: 6.4, 6.3 May 24, 2023

weaverryan added the ❄️ Feature Freeze Important Pull Requests to finish before the next Symfony "feature freeze" label May 24, 2023

weaverryan force-pushed the asset-mapper-jspm-esm branch from 4e90dbf to cdba8e7 Compare May 24, 2023 03:05

[AssetMapper] Change default importmap "provider" to JsDelivr+esm

de44614

stof requested changes May 24, 2023

View reviewed changes

src/Symfony/Component/AssetMapper/ImportMap/Providers/JsDelivrEsmImportMapProvider.php Outdated Show resolved Hide resolved

src/Symfony/Component/AssetMapper/Tests/ImportMap/ImportMapManagerTest.php Outdated Show resolved Hide resolved

carsonbot added Status: Needs Work and removed Status: Needs Review labels May 24, 2023

dunglas requested changes May 24, 2023

View reviewed changes

nicolas-grekas force-pushed the asset-mapper-jspm-esm branch 2 times, most recently from 70defcf to f624325 Compare May 24, 2023 15:36

nicolas-grekas approved these changes May 24, 2023

View reviewed changes

carsonbot added Status: Reviewed and removed Status: Needs Work labels May 24, 2023

nicolas-grekas force-pushed the asset-mapper-jspm-esm branch from f624325 to 2cf9176 Compare May 24, 2023 15:47

[AssetMapper] Fix wiring resolvers, send requests in parallel and use…

b530dc3

… readonly properties in MappedAsset

nicolas-grekas force-pushed the asset-mapper-jspm-esm branch from 2cf9176 to b530dc3 Compare May 24, 2023 15:56

weaverryan commented May 24, 2023

View reviewed changes

nicolas-grekas merged commit f64e38d into symfony:6.3 May 24, 2023
6 of 7 checks passed

weaverryan deleted the asset-mapper-jspm-esm branch May 24, 2023 19:20

fabpot mentioned this pull request May 27, 2023

Release v6.3.0-RC2 #50455

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[AssetMapper] Change default importmap "provider" to JsDelivr+esm #50408

[AssetMapper] Change default importmap "provider" to JsDelivr+esm #50408

weaverryan commented May 24, 2023 •

edited

Loading

carsonbot commented May 24, 2023

weaverryan commented May 24, 2023

stof commented May 24, 2023

dunglas left a comment

nicolas-grekas left a comment •

edited

Loading

nicolas-grekas May 24, 2023

nicolas-grekas May 24, 2023 •

edited

Loading

nicolas-grekas May 24, 2023

nicolas-grekas May 24, 2023

nicolas-grekas May 24, 2023

nicolas-grekas May 24, 2023

nicolas-grekas May 24, 2023

weaverryan left a comment

weaverryan May 24, 2023

nicolas-grekas commented May 24, 2023

[AssetMapper] Change default importmap "provider" to JsDelivr+esm #50408

[AssetMapper] Change default importmap "provider" to JsDelivr+esm #50408

Conversation

weaverryan commented May 24, 2023 • edited Loading

jsDelivr to the rescue

carsonbot commented May 24, 2023

weaverryan commented May 24, 2023

stof commented May 24, 2023

dunglas left a comment

Choose a reason for hiding this comment

nicolas-grekas left a comment • edited Loading

Choose a reason for hiding this comment

nicolas-grekas May 24, 2023

Choose a reason for hiding this comment

nicolas-grekas May 24, 2023 • edited Loading

Choose a reason for hiding this comment

nicolas-grekas May 24, 2023

Choose a reason for hiding this comment

nicolas-grekas May 24, 2023

Choose a reason for hiding this comment

nicolas-grekas May 24, 2023

Choose a reason for hiding this comment

nicolas-grekas May 24, 2023

Choose a reason for hiding this comment

nicolas-grekas May 24, 2023

Choose a reason for hiding this comment

weaverryan left a comment

Choose a reason for hiding this comment

weaverryan May 24, 2023

Choose a reason for hiding this comment

nicolas-grekas commented May 24, 2023

weaverryan commented May 24, 2023 •

edited

Loading

nicolas-grekas left a comment •

edited

Loading

nicolas-grekas May 24, 2023 •

edited

Loading