Dynamic import glob support

[guybedford]

Cases like import(`asdf/${}`), import('base/' + var) etc should all be turned into globbed dynamic import entry points for support in chunking.

We could possibly look at adding glob support to code splitting inputs in general here too, although we may have a problem in that input: ['asdf/*'] can work, while there isn't necessarily a way of doing that in an object - input: { '_': ['asdf/*'], named: 'entry' }...?

[lukastaegert]

Cases like import(asdf/${}), import('base/' + var) etc should all be turned into globbed dynamic import entry points for support in chunking

I agree that some kind of globbing would be certainly very powerful. But there are some basic questions that I struggle to answer:

• What is a globbed dynamic entry point? How are the imported files identified? Where should the generated chunks end up and what should be their names?
  • If we do not rewrite the dynamic imports, the file names need to match with the originally imported modules and end up in the same positions relative to the importing chunks
  • If we do rewrite them, this will seriously limit which kind of globbing is possible.
    This is not a problem for e.g. webpack which has a runtime to dynamically translate imports but it is considerably harder for rollup.
• Expanding on this, would should happen if modules 'a' and 'b' both dynamically import something like 'c' + variable, but 'a' is placed in chunk dist/chunk-a.js while 'b' is placed in chunk dist/nested/chunk-a.js?
• There are very valid reasons for NOT having dynamic imports be turned into globs by default. A CLI tool, for instance, may parse directories and then dynamically import certain files it finds.

At first glance, this looks to me like a feature that wants to add some convenience to something that is already and very reliably possible today. I.e. I could use the "named chunks" feature to create a number of possible chunks relative to my importing chunk and then dynamically import those. Maybe some full examples would be helpful to get a better idea of how such a feature might work.

We could possibly look at adding glob support to code splitting inputs in general

This, on the other hand, sounds very much doable and useful.

[guybedford]

@lukastaegert these are some great points.

So if you have import('x/' + sub') where the entry points map to different names in the output bundle, then we'd need to create a production-time mapping. If the mapping is predictable and "linear" we could just replace like import(import.meta.url + '../' + sub), and if it isn't we could replace with a dynamic map likeimport(import.meta.url + relmap['x/' + sub']).

What do you think of something like that?

Also auto-collecting globs by default is also a big question. An alternative would be ensuring users define the entry point paths explicitly: input: ['main', 'dynamics/*']. This might well be a good argument for supporting wildcards in the input object / array forms.

In the object form we could possibly do something like:

input: {
  'main': 'src/index',
  'dynamic-*': src/dynamics/*.js',
}

Where we support only a single non-deep wildcard that can act as its own substitution.

This stuff gets complex fast... but that's why I thought it worth putting out the wild ideas now - not exactly close to implementation here don't worry.

[lukastaegert]

then we'd need to create a production-time mapping. If the mapping is predictable and "linear" we could just replace like import(import.meta.url + '../' + sub), and if it isn't we could replace with a dynamic map like import(import.meta.url + relmap['x/' + sub'])

So we basically add a "mini-runtime" just for those files? I guess that would be acceptable. We would need to put some thought into how a file is handled that is imported both globbed-dynamically as well as synchronously by its explicit name.

For a first attempt, I would suggest to ALWAYS use the dynamic map to have the general case covered and then see what the most useful and easy optimizations are.

This might well be a good argument for supporting wildcards in the input object / array forms

As people keep asking about this, I think this is something we should definitely do! Ideally, we would also support ** globs. This of course would make the naming in the object form much harder. One way to address this could be to allow variables like [name], [ext] etc.

[guybedford]

So we basically add a "mini-runtime" just for those files? I guess that would be acceptable. We would need to put some thought into how a file is handled that is imported both globbed-dynamically as well as synchronously by its explicit name.

It's just a dynamic map for unknown input values - so far as we could predict inputs into the dynamic import we can make the rewrite more well-defined. Definitely starting with the map we could extend the optimization cases from there.

Ideally, we would also support ** globs

I'd be against ** globs if we can - if users have arbitrary depth to their entry points, they can still list those depths explicitly. Note patterns like packages/*/src/index.js would work fine here too.

One way to address this could be to allow variables like [name], [ext] etc.

We already support these replacements in the entryFileNames pattern which is used to convert from the "entry name" into the "entry file name". What we specify in the input object is the entry name that becomes the [name] pattern in the entryFileNames rule. This allows a single rule to name all entry points so switching to mjs is as easy as entryFileNames: '[name].mjs' or adding hashes with entryFileNames: '[name]-[hash].js'. So I'd be wary of conflating the replacement logic too much here.

[frank-dspeed]

Maybe we should also handle import(data:text/javascript,${str});
i have builded a esm-loader that uses that.

[LarsDenBakker]

I made a plugin for this: https://github.com/LarsDenBakker/rollup-plugin-dynamic-import-variables

[shellscape]

@LarsDenBakker any interest in having that part of this repo? totally cool if not, no worries!

[LarsDenBakker]

Yes that would be great!

[lukastaegert]

+1 from me as well, having something like this has been missing for quite some time!

[shellscape]

@LarsDenBakker wonderful. We don't have a formal process for this in place (because it doesn't happen often enough to have one). I've sent you an invite to the plugin maintainers team and created the migrate/dynamic-import-vars. Please take a look at the other migrate PRs from when we moved all of the plugins here, and go from there. Once you're happy with the branch, PR and we'll go from there!

I'd humbly suggest shortening the name a bit to dynamic-import-vars, but that's not a requirement by any means.

[milahu]

a more generic solution are compile-time expressions
https://github.com/egoist/vite-plugin-compile-time

in my case, i also need the filenames of the globbed imports

// codegen-files.js

import glob from "tiny-glob"
export default async function () {
  const prefix = "../src";
  const files = await glob(prefix + "/**/*.{ts,js}", {
    cwd: __dirname,
    filesOnly: true,
  });
  const object = "filesystemFiles";
  const str = JSON.stringify;
  const importCode = files.map(path => {
    // remove prefix + extension
    const name = path.slice(prefix.length).replace(/\.[a-z0-9]+$/i, "");
    return `${object}[${str(name)}] = () => import(${str(path)});`;
  }).join("\n");
  return {
    //data: files,
    code: importCode,
  };
}

// index.js

/** @type {Record<string, () => any>} */
const filesystemFiles = {};
import.meta.compileTime("./codegen-files.js");

for (const [file, importFile] in Object.entries(filesystemFiles)) {
  if (needFile(file)) {
    const fileModule = await importFile();
    useFile(fileModule);
  }
}

edit: vite != rollup ...

[frank-dspeed]

Out of my view globing in any parts like input would hurt essential at maximum we can go for generateBuild hook for assets globs and buildStart hook to emit the glob results we should never no where in the core try to put globs in that breaks ecmascript.

and yes vite is rollup the first 3 steps of a single file and maybe even multiple are the same. @milahu can be seen as offtopic as he used import meta helpers no one knows where that come from maybe that's something vite added.