Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

How do i mark some modules to external? #144

Closed
garrydzeng opened this issue Dec 8, 2017 · 95 comments
Closed

How do i mark some modules to external? #144

garrydzeng opened this issue Dec 8, 2017 · 95 comments

Comments

@garrydzeng
Copy link

@garrydzeng garrydzeng commented Dec 8, 2017

Choose one: is this a 馃悰 bug report or 馃檵 feature request?

馃檵 feature request

馃 Expected Behavior

Don't include external module in bundled file everywhere.
Like rollup globals option.
https://rollupjs.org/#big-list-of-options

馃實 Your Environment

Software Version(s)
Parcel 1.0.3
Node 9.2.0
npm/Yarn Yarn 1.2.1
Operating System Windows 10
@devongovett
Copy link
Member

@devongovett devongovett commented Dec 20, 2017

Can you elaborate on the behavior you'd like to see here? I'm not sure how the option as documented by rollup is useful. Why not just use the global variable? Why is it an import at all?

@garrydzeng
Copy link
Author

@garrydzeng garrydzeng commented Dec 20, 2017

@devongovett

Let me take React as an example.

I can use React as global variable in my project, but many third-party library imported React in their code and I can't control that, so, if I include React through a CDN, like <script src="//cdnjs.cloudflare.com/ajax/libs/react/16.2.0/umd/react.production.min.js"></script> and the bundle in my index.html, then browser has to download React twice.

  • one copy in the bundle
  • one from CDN

In order to avoid this problem, I have to import React in my project. If Parcel can replace React as global variable when it handling third-party library, then it can reduce bundle size & I can use CDN to speed up my website. React can be any library also.

I made an example: parcel-example.zip.
Hope this help!

@devongovett
Copy link
Member

@devongovett devongovett commented Dec 23, 2017

I see. This seems somewhat related to aliasing actually. See #25.

@MalcolmDwyer
Copy link

@MalcolmDwyer MalcolmDwyer commented Feb 12, 2018

Another vote for this issue... I really like what you've done here with Parcel. It was super-easy to get my project 95% set up. This issue, (external resources), along with dev-server proxying are the two things holding me back.

Motivation for both is that I'm developing components that will be used within a larger application. There are external dependencies (that I don't want packaged up in the component build), and there are external APIs I want to hit while developing. In both cases, these are things that will be there when the project is deployed, but which are not a part of this component.

garrydzeng mention Rollup's "globals" option. Webpack similarly has "externals" (https://webpack.js.org/configuration/externals/).

You mentioned babel aliasing. That, I think, is a separate issue, because it's after the packager has decided to include a file, you can tell babel an alternate path to get it. There doesn't appear to be a way to tell babel that it should be ignored entirely. (At least not in a way that will keep other things happy). The "external" concept would really need to be configured at the packager.

I'd be happy to help out and contribute code to solving these issues, but given the 'zero-configuration' mantra, it's not clear to me where or how either could/should be done in Parcel. If you have ideas along those lines, please let me know. --Thanks

@starkwang
Copy link

@starkwang starkwang commented Feb 27, 2018

+1 on this issue. The aliases implemented in #850 can not solve this issue. Just like globals in Rollup and externals in webpack, Parcel also need an option to map modules to a global value.

I'd like to help with this issue if needed : )

@derolf
Copy link

@derolf derolf commented Mar 4, 2018

I have a patched version that allows external scripts. The idea is to mark scripts as 鈥渃opy鈥 and provide a mapping of globale to modules that. But that still is kind of configuration...

@davidnagli
Copy link
Member

@davidnagli davidnagli commented Mar 4, 2018

@starkwang Cool! I鈥檇 be great if you can take this issue 馃槂

@DeMoorJasper
Copy link
Member

@DeMoorJasper DeMoorJasper commented Mar 4, 2018

@derolf @davidnagli Perhaps change the aliasing a slight bit and add something like this:

aliases {
  "react": false // This will ignore the package
}

This would cause it to skip without adding extra complex configurations. (Of course we still have to implement it and i'm not sure if it's sorta allowed by the standards)
This behaviour would be kinda similar to how browser.fs === false works for fs resolves

@derolf
Copy link

@derolf derolf commented Mar 4, 2018

I have a proposal to extend the "alias" syntax to support externals:

"<module>": "<file>!<evaluated export term>"

Example (for cesiumjs):

"cesium": "./node_modules/cesium/Build/Cesium/Cesium.js!Cesium"

Semantics:

  • copy ./node_modules/cesium/Build/Cesium/** into the dest folder
  • lazy load or include in bundle (depending on how it is used)
  • create fake module "cesium" with cesium.exports = Cesium

Usage:
HTML inline load: <script src="cesium"></script>

@DeMoorJasper
Copy link
Member

@DeMoorJasper DeMoorJasper commented Mar 4, 2018

@derolf isn't this different than the described issue? The issue is about using external packages from cdn's wouldn't this just be lazyloading everything from the local server or cdn?

@derolf
Copy link

@derolf derolf commented Mar 4, 2018

@DeMoorJasper

For CDN:

"cesium": "http://cdn.xyz.com/foo/bar/Cesium.js!Cesium"

could work the same way. But instead of copying into dest folder it's is directly linked.

I think the main idea is to somehow refer to "prebuilt" folders -- disable parcel parsing -- and link them to a fake module.

So the exclamation mark disables parcel parsing of the content and the term afterwards creates the export directive for the fake module.

@woubuc
Copy link

@woubuc woubuc commented Mar 5, 2018

@derolf I think the ! syntax is fairly non-obvious, at least I haven't seen it used before. Is this 'standard' across similar implementations? Why not just use the key?

@derolf
Copy link

@derolf derolf commented Mar 5, 2018

@woubuc So, if you look at webpack's external, you need a key plus which global symbol to bind to that key. So, we need to tell parcel which symbol to bind to the fake module.

Example: three-js exports a global symbol called THREE, so the syntax would be:

"three": "https://cdnjs.cloudflare.com/ajax/libs/three.js/90/three.min.js!THREE"

This follows the same approach in webpack.

@MalcolmDwyer
Copy link

@MalcolmDwyer MalcolmDwyer commented Mar 5, 2018

@DeMoorJasper 's <PackageName>: false makes sense to me (but misses renaming... which may be important in certain cases).
I don't see the advantage of @derolf's <PackageName>: <CDN path>:!<import name>, unless it's solving a different problem than what I was talking about.

Could the webpack externals concept be followed directly, except inside package.json. Then it's a 1-to-1 with a well-known feature, and keeps the ability to rename the import.

package.json:

  "dependencies": { },
  "parcel": { /* Or should externals be at the top level? */
    "externals": {
      "MyProjectConfig": "config",
      "react": "react"
    }
  }
}

Unlike the aliases: {packageName: false}, this allows for renaming. You can import from the proper name of the package (as if you were really packaging it up), but still have the code get linked into whatever the actual named variable is in the CDN/inline script.

My deployed index.html/jsp that pulls in the parceled project...

   <script>
      config = {
        someValue: 123
      }
   </script>
   <script src="://some-cdn/react.min.js"></script>
   <script src="./build/parceled-up-project.js></script>

And finally... some random source file in my project:

import MyProjectConfig from 'MyProjectConfig'
import React, { Component } from 'react'

For both those imports, the packager can find the 'from' in the externals list, and know that it doesn't need to actually fetch/import anything. And the actual variable names can get assigned with the desired renaming.

Something like this makes the most sense to me... but it's obviously getting away from 'zero-config', so I assume it would need some consensus before building it in. What do you think?

@derolf
Copy link

@derolf derolf commented Mar 5, 2018

Package.json is not available to the parser. Because resolution is done afterwards.

So, you need two things:

  • disable parcel鈥檚 parsing if you pull in a prebundled script
  • link a module to an exported variable
@derolf
Copy link

@derolf derolf commented Mar 5, 2018

@MalcomDwyer okay, I can implement your 鈥渆xternals鈥 in package.json proposal and create module stubs the same way that Webpack does.

But, what is a good syntax to tell parcel to NOT bundle a script that is already in the public folder?

Like: <script parcel-ignore src=鈥榓bc.js>

@MalcolmDwyer
Copy link

@MalcolmDwyer MalcolmDwyer commented Mar 5, 2018

But, what is a good syntax to tell parcel to NOT bundle a script

In my example, that "deployed index.html/jsp" would not even be something that parcel ever sees...

I have my frontend project codebase that I use to develop, and test the javascript project. Then in a separate downstream project, I just pick up the bundle.js (src="./build/parceled-up-project.js in my example above) as a pre-built entity. It is in that downstream file that those external imports actually get resolved. (So they are external to parcel, but are part of the downstream project).

So I would probably have parcel building the real bundle.js (which ignores those externals), but also for development, I'd have parcel build a demo/index.html which would have the same imports and should include those files. I can't think of a time I'd be pointing parcel at an html with <script src="x"> and not want it to import "x".

@derolf
Copy link

@derolf derolf commented Mar 5, 2018

I start bundling from a central HTML file and need to pull in stuff to parcel and stuff that is prebuilt (three js and cesium js).

@DeMoorJasper
Copy link
Member

@DeMoorJasper DeMoorJasper commented Mar 5, 2018

Package.json is not available to the parser. Because resolution is done afterwards.

You shouldn't wait this long right?
You just append the externals to the resolver, this way u can detect it before it even gets to the parser

<script parcel-ignore src=鈥榓bc.js>

No big fan of this, it would add a new sort of config to the ecosystem and limit the abilities it has, this seems to be more a use-case for the false key I described in my first comment on this issue, as this would extend beyond just html (adding the possibility to ignore in JS, for example optional dependencies parcel tries to resolve)

@derolf
Copy link

@derolf derolf commented Mar 5, 2018

As far as I saw, parsing is done before resolving.

So, how can I tell the parser of the index.html not to bundle a script, but just leave it untouched.

@DeMoorJasper
Copy link
Member

@DeMoorJasper DeMoorJasper commented Mar 5, 2018

In the current codebase u should look at asset.addUrlDep... and the Resolver as those are the only places we use to actually resolve assets.

I'll try to make it a bit more clear, the parser.getAsset() never gets called if the dependency never gets added to the asset in the first place. The package is known to the parent asset, so you have full access to package.json

@derolf
Copy link

@derolf derolf commented Mar 5, 2018

@DeMoorJasper I looked a lot at the code this weekend and found the asset.addUrlDep. The problem is that the parsing of the asset, processing, and generating the output is done in a Worker. This Worker has NO access to the resolver, but just creates put dependency-links to the bundles that will contain these dependencies into the processed files.

Then later, the bundler picks up the queued dependencies and resolves (including aliases) them and starts another Worker process.

Check this Bundler: loadAsset(asset)

async loadAsset(asset) {
... FIRST PROCESS
    let processed = this.cache && (await this.cache.read(asset.name));
    if (!processed || asset.shouldInvalidate(processed.cacheData)) {
      processed = await this.farm.run(asset.name, asset.package, this.options);
...
    }
... THEN RESOLVE
    // Resolve and load asset dependencies
    let assetDeps = await Promise.all(
      dependencies.map(async dep => {
...
          let assetDep = await this.resolveDep(asset, dep);
...
      })
    );

...
}

To fix this, we would need to expose the resolver to the Worker and let resolution happen there???

@derolf
Copy link

@derolf derolf commented Mar 5, 2018

Ah, you mean Assert.pkg? You are right, it's having full access to the package.json!

@DeMoorJasper
Copy link
Member

@DeMoorJasper DeMoorJasper commented Mar 5, 2018

I'm pretty sure if the dependency never gets added, the bundler never picks them up anyway? The resolver changes would just be in case the bundler adds some dependencies.
Now that I looked at it resolver changes aren't even neccessary unless u wanna move the responsibility to the bundler, I haven't tested any of the things i say i'm suggesting as a fix.

And the code sample you show makes sense but doesn't have much to do with the ignore or this issue. It will never get to loadAsset if the asset is ignored, as that happens on the loadAsset of the previous/parent asset.

And if it's really necessary u could spin up a resolver per worker similar to how the parser is being used inside workers, although it would be nice to prevent this.

EDIT: Yes i do mean Asset.pkg (of the parent of the ignored/external package)

@derolf
Copy link

@derolf derolf commented Mar 5, 2018

So, which syntax do we follow now? An external needs TWO things: where is the file located and what does it export.

Example with cesium js:

"externals": {
 "cesium": {
   "exports": "Cesium", // --> this generates a module stub with cesium.exports = Cesium;
   "file": "node_modules/cesium/Build/Cesium/Cesium.js", // ---> that is the file to be src for the script
   "copy": "node_modules/cesium/Build/Cesium", // ---> that tells parcel to copy (or symlink) all that content into the public dir
 }
}

"copy" here is important since Cesium.js includes other files from that folder. So, the WHOLE folder needs to be present!

Makes sense?

@maximal
Copy link

@maximal maximal commented Oct 16, 2019

+1 here

This keeps me from switching to Parcel on about 5 projects.

@paramsinghvc
Copy link

@paramsinghvc paramsinghvc commented Oct 16, 2019

Love parcel but let's face it, it's not just as mature as Webpack yet!

@FlorianRappl
Copy link

@FlorianRappl FlorianRappl commented Oct 16, 2019

I'm not sure about that. Webpack's code is (or was when I last looked at it) much more horrible. Additionally, the ecosystem of Webpack is quite fragmented. I lost interest in upgrading Webpack at some point - because either half of the plugins did not work any more at all, had some serious side-effects, or produced garbage.

With Parcel I never had a problem (but on the other side the API was not broken as often as with Webpack - I'm a little bit afraid of v2 for this reason).

@davedoesthings
Copy link

@davedoesthings davedoesthings commented Oct 16, 2019

Webpack may be more mature, but maturity !== simplicity. Parcel is a far, far simpler solution to the toooling needs of many, and this issue, once resolved, will unblock a lot of people who are waiting to jump ship for good reason.

@youknowriad
Copy link

@youknowriad youknowriad commented Oct 16, 2019

I wonder if it's the best place to discuss parcel vs webpack for folks following this issue :P

@FlorianRappl
Copy link

@FlorianRappl FlorianRappl commented Oct 16, 2019

Actually I don't think it's that difficult to do. We do that in Piral where we use Parcel as our build system integrated in our CLI. For us we need to strip out shared dependencies from frontend modules (called Pilets).

What we do could be (i.e., is definitely) considered a hack (and obviously I wish that there may be a simpler alternative, which may exist with Parcel 2), but it could easily be applied / transported in a plugin.

If you guys are interested I guess I could try to craft a simple plugin on the weekend. Only thing to discuss upfront is what packages to mark as external.

Options I see: peerDependencies (this is what we do in Piral essentially - though its a bit more complicated than that), a new key in package JSON (e.g., externalsPackages) or in a dedicated config / rc file.

Any thoughts?

@devongovett
Copy link
Member

@devongovett devongovett commented Oct 16, 2019

Seems like maybe there's two separate but related features here:

  1. Excluding a module from a bundle, but leaving the import or require there. Use case would be for e.g. libraries with peer dependencies, builtin modules in node/electron, etc. Maybe this could be solved by excluding peerDependencies.
  2. Mapping an import to a global variable that is available somehow. Use case would be for e.g. libraries from a cdn. This might need separate configuration, e.g. with aliases.

See also #3305

@FlorianRappl
Copy link

@FlorianRappl FlorianRappl commented Oct 20, 2019

So I hacked this together: https://www.npmjs.com/package/parcel-plugin-externals

Don't know if its useful. If there is some use, please give feedback. Thanks! 馃嵒

@subhero24
Copy link
Contributor

@subhero24 subhero24 commented Oct 25, 2019

I created a pull request to support this without a plugin

@thinkloop
Copy link

@thinkloop thinkloop commented Oct 25, 2019

You are officially a fullhero now 馃憤

@akrizs
Copy link

@akrizs akrizs commented Nov 1, 2019

Guys! You are making parcel the easiest tool in the toolbelt to use! That is amazing! Thanks alot!

@mikestopcontinues
Copy link

@mikestopcontinues mikestopcontinues commented Dec 26, 2019

What's the status of this issue? As per @devongovett 's first use-case:

Excluding a module from a bundle, but leaving the import or require there. Use case would be for e.g. libraries with peer dependencies, builtin modules in node/electron, etc. Maybe this could be solved by excluding peerDependencies.

I'm deploying to Lambda, and I don't need to include the tremendous aws-sdk in my package. I just need a way to let require('aws-sdk') access the global available on Amazon's servers.

@cliffordp
Copy link

@cliffordp cliffordp commented Feb 13, 2020

@FlorianRappl #144 (comment)
Saved my bacon to be able to use Parcel for a WordPress plugin that uses jQuery (which comes from WP itself)
THANK YOU!

@cliffordp
Copy link

@cliffordp cliffordp commented Mar 19, 2020

@FlorianRappl

Any idea how to glob-exclude? I don't want these:
image

@FlorianRappl
Copy link

@FlorianRappl FlorianRappl commented Mar 19, 2020

Hi @cliffordp - please create issues at https://github.com/FlorianRappl/parcel-plugin-externals such that they have increased visibility for other users of the plugin.

Have you tried https://github.com/FlorianRappl/parcel-plugin-externals#dynamic-dependency-resolution with a rule factory?

I can imagine that (taking only the @wordpress/* packages into consideration)

const rx = /node_modules\/@wordpress\/(.*?)\//;

module.exports = function(path) {
  const result = rx.exec(path);

  if (result) {
    const package = result[1];
    return `@wordpress/${package} => require('@wordpress/${package}')`;
  }

  return undefined;
};

could help here.

@DeMoorJasper
Copy link
Member

@DeMoorJasper DeMoorJasper commented Mar 21, 2020

#4072 implemented this, for additional ways to exclude modules feel free to open an RFC to open discussion on this

@cliffordp
Copy link

@cliffordp cliffordp commented Mar 21, 2020

@DeMoorJasper that didn't work for me (unless the "library" bit needs to be customized for my use case, in which I'm unsure how-to):

image

@FlorianRappl
Copy link

@FlorianRappl FlorianRappl commented Mar 22, 2020

@cliffordp I think this is a Parcel v2 addition - for Parcel v1 I guess you'll need to stay with the parcel-plugin-externals.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.