Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Custom loader that invokes other loaders #860

Closed
mlindsten opened this Issue Mar 5, 2015 · 15 comments

Comments

Projects
None yet
5 participants
@mlindsten
Copy link

mlindsten commented Mar 5, 2015

Hi folks!

I'm quite new to webpack (I've been experimenting with it in my spare time for 2-3 months now), but I've already written a couple of plugins and some loaders (maybe I'm misusing it since I need to modify how it works out of the box, but anyway...).

I'm currently working on a loader that "combines" different kinds of content (html, css, etc) based on a specific source file (a vue-component file, if you're curious). After reading blocks of different content from the file, I want my loader to invoke the other loaders defined in the webpack options for each content block, just as if they were separate files of their own, and then continue processing (i.e. combining) the results. Do you follow?

Currently my loader works by tricking webpack to think there are different files, but this is messy and it also doesn't work with the dev-server, so I want to improve it by directly invoking the other loaders from my loader.

I've been digging through the _compiler and _compilation objects available in the loader context, but I feel a little lost. So, assuming my loader is invoked for path/to/myfile.vue, how can I, from my loader, invoke all the loaders that would be invoked for the hypothetical file path/to/myfile.less, providing the file contents as a string and getting the resulting script back as a string?

@sokra

This comment has been minimized.

Copy link
Member

sokra commented Mar 5, 2015

the hypothetical file path/to/myfile.less, providing the file contents as a string and getting the resulting script back as a string?

You cannot fake files from a loader. Instead you should try write a loader that extract a part of the file and use something like the multi-loader to configure all parts in the webpack.config.js.

You propably need to modify the multi-loader a bit to take a object which specifies the exports

{
    module: {
        loaders: [
            {
                test: /\.vue$/,
                loader: multi({
                    html: "html-loader!vue-loader?html",
                    css: "style-loader!css-loader!vue-loader?css",
                    component: "vue-loader?js"
                })
            }
        ]
    }
}
@mlindsten

This comment has been minimized.

Copy link
Author

mlindsten commented Mar 5, 2015

You cannot fake files from a loader.

Well, I'm not exactly trying to "fake files". I have extracted the different parts of the source file into strings, one containing JavaScript, one HTML and another one LESS code (as an example). What I want to do now is to pass each string through the regular chain of loaders defined in webpack options as if they were files of their own, meaning that only loaders matching path/to/myfile.js, path/to/myfile.html and path/to/myfile.less, respectively, will be invoked. Choosing which loaders to invoke is not the problem. I can pick up the loader configuration from context.options and do the matching. The problem is that I don't know how to invoke the loaders. I've been looking at context._compiler.resolvers.loader.resolve, but I haven't explored it any further and I'm not sure if it's a dead end.

Are you saying it's not possible to invoke another loader from inside a loader?

Instead you should try write a loader that extract a part of the file and use something like the multi-loader to configure all parts in the webpack.config.js.

That's kind of how I've solved it now (without knowing about the multi-loader), but it feels clumpsy to have to run the loader multiple times and I also don't like that it produces extra "gluing" output.

@sokra

This comment has been minimized.

Copy link
Member

sokra commented Mar 5, 2015

Are you saying it's not possible to invoke another loader from inside a loader?

You can either do this by emitting js code that calls require(...) with that loader. Or you can use this.loadModule(request, callback) with compiles a module and callbacks with the result.


Try to not use this._compiler, this._compilation or this.options.


Try to design your loaders to use chaining. I. e. your loader shouldn't know how parts of the vue component are processed. It should return the parts and give the user the opportunity to define how the parts are processed.

Here is an example:

It have files that can contain multiple parts separated by ---.

exports.answer = 42;
---
body { background: red; }

I can now write a loader that does exactly one task: "get one of the parts of the file". I name it multipart-loader and it takes a number as query argument.

i. e. require("multipart?0!./file.js+css") runs only the component. require("style!css!multipart?1!./file.js+css") runs the css part.

A configuration like this allow to require("./file.js+css"), which runs both:

{ test: /\.js+css$/, loader: multiLoader(
  "style!css!multipart?1",
  "multipart?0"
)}

And with this flexiblity I give the user the choice to change the behavior to their needs:

{ test: /\.js+less$/, loader: multiLoader(
  "style!css!less!multipart?1",
  "multipart?0"
)}
{ test: /\.js+css$/, loader: multiLoader(
  ExtractTextPlugin.extract("style", "css!multipart?1"),
  "multipart?0"
)}

The multipart-loader can internally call this.loadModule("!!" + require.resolve("./getParts") + "!" + remainingRequest, ...) to get an array of all parts. Grab the correct part and pass (return) it to the next loader.

@mlindsten

This comment has been minimized.

Copy link
Author

mlindsten commented Mar 5, 2015

I. e. your loader shouldn't know how parts of the vue component are processed.

I see your point! However, the format of the source file (the Vue component) specifies the type of each block and I want the contents of the blocks to pass through the regular loaders (defined in module.loaders) based on this type. The format looks like this (notice the lang attribute):

<script lang="coffee">
    ...
</coffee>
<template lang="jade">
    ...
</template>
<style lang="less">
    ...
</style>

The context.loadModule(...) method is interesting, though! If I (against your recommendation) read the loaders configuration from context.options, could I use this method to do what i want, i.e. pass each block through the corresponding set of loaders?

@sokra

This comment has been minimized.

Copy link
Member

sokra commented Mar 5, 2015

You could still allow the user to configure it in webpack.config.js:

{ test: /\.vue$/, loader: vueLoader({
  coffee: "coffee-loader",
  jade: "jade-loader",
  css: "style!css",
  less: "style!css!less"
}) }

could I use this method to do what i want, i.e. pass each block through the corresponding set of loaders?

You cannot pass the blocks into it. It only runs a request, so you cannot pass it stuff...

I think you can solve it easily by generating require(...) calls to the parts.

@mlindsten

This comment has been minimized.

Copy link
Author

mlindsten commented Mar 5, 2015

You cannot pass the blocks into it. It only runs a request, so you cannot pass it stuff...

Ok. So, one way or another, I have to through my loader once for each block (to extract that block)? Will that also read the file multiple times, or does webpack cache the file data in memory?

@sokra

This comment has been minimized.

Copy link
Member

sokra commented Mar 5, 2015

Will that also read the file multiple times, or does webpack cache the file data in memory?

The read is cached.

But you can implement it in a way that splitting is only done once:

The multipart-loader can internally call this.loadModule("!!" + require.resolve("./getParts") + "!" + remainingRequest, ...) to get an array of all parts. Grab the correct part and pass (return) it to the next loader.

@mlindsten

This comment has been minimized.

Copy link
Author

mlindsten commented Mar 5, 2015

The multipart-loader can internally call this.loadModule("!!" + require.resolve("./getParts") + "!" + remainingRequest, ...) to get an array of all parts. Grab the correct part and pass (return) it to the next loader.

Hmm, I don't really see how that only processes the file once. I think I need to read up on the loadModule(...) method and the syntax for loaders in require calls. Thanks for your help!

@sokra

This comment has been minimized.

Copy link
Member

sokra commented Mar 5, 2015

Oh, I should say when calling loadModule(...) multiple times with the same request, the module is only constructed once. This means the loader only runs once.

@VictorBlomberg

This comment has been minimized.

Copy link

VictorBlomberg commented Mar 10, 2015

@mlindsten Hi! Have you had any success? Over at issue #547, we are looking for something similar for Web Components/Google Polymer.

Sorry to bother you, but since we are quite lazy, and want the same thing, it would be a shame if you are sitting on stuff and willing to share. Maybe it would even be relevant to join efforts?

@mlindsten

This comment has been minimized.

Copy link
Author

mlindsten commented Mar 13, 2015

@VictorBlomberg Sorry about the late reply! I've been up to my ears in work lately and I still am, so I haven't had the time to improve my loader. It works ok as it is now, but I'm still not really satisfied with it.

It reads the loaders from context.options (despite the dissuasion from sokra) and creates enhanced require statements for the different parts. The source file is parsed only once and the different parts are kept temporarily in a separate cache. It also handles bootstraping of Vue components so that I can use them directly as entries.

Unfortunately, I don't have a GitHub repo for the loader (since I wrote it to be used in a larger project), but I can send you the code if you're interested. Oh, and I should say that it's written in ES6 so you'll need a transpiler (I use 6to5/Babel myself).

@sjoerdvisscher

This comment has been minimized.

Copy link

sjoerdvisscher commented May 7, 2015

@sokra I took a shot as implementing this as you suggested. https://github.com/Q42/vue-multi-loader
I wonder if I got your loadModule suggestion correct. Am I supposed to JSON.stringify the result, and then exec it in the loadModule callback, or is there a better way?
Also, do you think it would be possible to create source maps?

@sokra

This comment has been minimized.

Copy link
Member

sokra commented May 8, 2015

Looks great.

A few notes: Try to use loaderUtils.getRemainingRequest(this) instead of this.resource to make the loader pipeable. Instead of 'require(' + JSON.stringify use loaderUtils.stringifyRequest, it removes absolute paths.

SourceMaps should be possible when the parser allows SourceMaps. You can embed the SourceMap in the parser result and return it in the selector.

sjoerdvisscher added a commit to Q42/vue-multi-loader that referenced this issue May 13, 2015

@bebraw bebraw added the question label Nov 15, 2015

@bebraw

This comment has been minimized.

Copy link
Member

bebraw commented Nov 15, 2015

Looks like there's https://github.com/vuejs/vue-loader now.

@mlindsten Safe to close?

@mlindsten

This comment has been minimized.

Copy link
Author

mlindsten commented Nov 15, 2015

@bebraw, sure!

I haven't had the time to try the vue-loader so I don't know if it does what I want, but you can close anyway since I've not been working on this since my last comment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.