Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Official Asset Pipeline #272

Closed
zachleat opened this issue Oct 8, 2018 · 34 comments
Closed

Official Asset Pipeline #272

zachleat opened this issue Oct 8, 2018 · 34 comments
Labels
enhancement needs-votes

Comments

@zachleat
Copy link
Member

@zachleat zachleat commented Oct 8, 2018

Surfaced in #270.

Maybe included as a separate starter project, plugin, and/or with the eleventy-base-blog project

@zachleat zachleat added enhancement needs-votes labels Oct 8, 2018
@zachleat
Copy link
Member Author

@zachleat zachleat commented Oct 8, 2018

This repository is now using lodash style issue management for enhancements. This means enhancement issues will now be closed instead of leaving them open.

The enhancement backlog can be found here: https://github.com/11ty/eleventy/issues?utf8=%E2%9C%93&q=label%3Aneeds-votes+sort%3Areactions-%2B1-desc+

Don’t forget to upvote the top comment with 👍!

@zachleat zachleat closed this as completed Oct 8, 2018
@kleinfreund
Copy link
Contributor

@kleinfreund kleinfreund commented Oct 8, 2018

In my opinion, Eleventy’s job is to expose the relevant pieces of content at appropriate stages of transformation so that available tooling can be used via their Node packages.

There is already some interesting examples of how to realize this is in the docs (e.g. compressing CSS).

It would be interesting to figure out how to use Eleventy with JavaScript bundlers like Parcel.js or Webpack. Can we somehow offer a simple way of using Eleventy and; for example, Parcel.js without resorting to starting both Eleventy and Parcel is separate terminal windows in parallel?

I think Eleventy has great potential in solving this build tool pain points that a lot of static site generators bring with them.

@edwardhorsford
Copy link
Contributor

@edwardhorsford edwardhorsford commented Oct 9, 2018

I wouldn't have considered picking up Eleventy were it not for the Eleventyone project. I'm not a developer, so though I'm sure it's possible to add yourself, it wasn't something I wanted to attempt. I've used grunt before, but again, didn't set it up to start with.

If it's helpful, here's some things I wanted:

My must haves:

  • SASS compilation
  • Asset concatenation / uglificiation
  • Moving assets around
  • Cleaning out the build directory / ability to start 'clean'
  • Something I understood enough that I could hook in my own code.

For me, gulp has worked well. I can have different scripts for dev and production so different things get built - such as uncompressed / sass-maped sources on dev, and fully minified sources on prod. I don't concatenate all files together - assets that may not be needed, or aren't needed by most pages are kept and served separately. Gulp made this easy.

It's possibly related to my background, but I prefer the idea of css concatenation happening separately from templates.

A possible useful starter might be a tutorial / guidance on adding a build pipeline. I suspect you want to keep the base repo rather agnostic about this - this is where the tutorial section could really help I think.

@jevets
Copy link

@jevets jevets commented Oct 11, 2018

I'd certainly vote for this to be outside the core, as an explicit opt-in or just a sample/starter project.

I've had luck with laravel-mix in porting one project over (it was already using mix). I just run eleventy --watch in parallel with laravel-mix's watch in the same npm script. Eleventy rebuilds the HTML, and laravel-mix writes directly to the dir.output path. One terminal window.

I've been working on setting up this kind of workflow with webpack v4 (mix is still at v3), but I've also been playing with parcel.js. I'll post a repo once I have something reasonable to share. It would probably bypass browserify completely, but I'd still want live reloading and watchers for all files.

@itlackey
Copy link

@itlackey itlackey commented Oct 23, 2018

I'm looking into this as well. 11ty works well with assets in the site content (custom scripts and styles) but how to manage npm dependencies is proving difficult. For example, I added bootstrap to package.json now I need my pipeline to bundle all of bootstraps assets with mine. Currently I'm looking at calling browserify or webpack from gulp but looking for a simple answer if someone has solved it already.

Thanks!

@jevets
Copy link

@jevets jevets commented Dec 6, 2018

Worth mentioning this here, please upvote if you like this idea of configuring eleventy to watch other globs:
#333

@paulshryock
Copy link
Contributor

@paulshryock paulshryock commented Jan 10, 2019

A possible useful starter might be a tutorial / guidance on adding a build pipeline. I suspect you want to keep the base repo rather agnostic about this - this is where the tutorial section could really help I think.

I think it would be great to include documentation about how to include some different asset pipelines easily (and maybe go so far as to provide those as plugins), but as others have said, it'd be nice to not include those by default in core. I really like it being "Bring your own ____" (markup/styles/asset pipeline/cms/etc.)

@jevets
Copy link

@jevets jevets commented Jan 24, 2019

I managed to get something working pretty well with webpack4 and eleventy.

  • Watches js, css, 11ty template files
  • Auto refresh on change of any of the above files
  • Builds minified js/css
  • Babel support
  • Uses eleventy --serve and webpack --watch

I tried getting it working with webpack-dev-server, but I couldn't manage to get both the 11ty side and the webpack side to recognize each others' changed files — auto refresh wouldn't work.

I'll probably put up a fresh repo soon and link to it here, maybe w/ a brief tutorial about this approach. I could make that repo as a starter project, too/instead.

A plugin of sorts would be nice as a simple drop-in, yet I don't know how I'd approach it as a standalone plugin. (@zachleat any thoughts on how to approach this?) It currently uses package.json scripts, but at least it'd give some people an idea how to approach using it in their own projects.

Looking to gauge interest and get all your opinions/thoughts on this as starter project vs a drop-in plugin type thing.

I think it makes more sense as a starter project or just a demo/tutorial. A drop-in plugin may end up hiding too much, getting us further away from "bring your own x".

I'm not married to webpack (I'm certainly no webpack pro), so I may try another bundler like parcel.js and see if it's any easier.

@robdodson
Copy link

@robdodson robdodson commented Feb 5, 2019

One thing that I think is common in the webpack space is to give files cache busting hashes and use the HTMLWebpackPlugin to make sure the proper hashes get included in the html files:

For example:

<html>
  <head>
    <!-- produces main.52l3kj51oij6.css -->
    <link rel="stylesheet" href="<%=htmlWebpackPlugin.files.chunks.main.css %>">
  </head>
  <body>
    <div>Hello, world!</div>
    <!-- produces bundle.lfkl2ekjlk.js -->
    <script src="<%= htmlWebpackPlugin.files.chunks.main.entry %>"></script>
  </body>
</html>

@jevets did you do any of this with your eleventy integration?

@jevets
Copy link

@jevets jevets commented Feb 11, 2019

@robdodson No, I didn't get that far but am aware of the need. May ask for some input from you if you're down. Would love to have some others help on this.

(Been away for a few weeks, will try to post some code this week.)

@chrisdmacrae
Copy link
Contributor

@chrisdmacrae chrisdmacrae commented Feb 16, 2019

https://github.com/chrisdmacrae/eleventy-starter-parcel

@dweidner
Copy link

@dweidner dweidner commented Feb 25, 2019

One thing that I think is common in the webpack space is to give files cache busting hashes and use the HTMLWebpackPlugin to make sure the proper hashes get included in the html files:

@robdodson You could use something like webpack-manifest-plugin or webpack-asset-manifest to generate a json file that maps your entry points to the generated files.

// Example manifest.json
{
  "main.js": "assets/js/main.12345645423.js"
}

In eleventy a custom shortcode or filter could than be used to lookup the generate file name from the json file.

@Ryuno-Ki
Copy link
Contributor

@Ryuno-Ki Ryuno-Ki commented Mar 17, 2019

I'd prefer to be bundler agnostic (yes, I don't like WebPack, but prefer Rollup).

@jevets
Copy link

@jevets jevets commented Mar 20, 2019

I've been using a stylesheet.11ty.js template file to compile CSS (via postcss in my case).
I have an _includes/actual-styles.css file, and when using eleventy --serve [| --watch], both files update automatically on change (for free).

With this route I get to avoid pulling in any additional packages (webpack, gulp, parcel, etc.).

It's been working very well for me lately, but I haven't yet used this route for sites heavy on client-side JS.

Something like this:

// stylesheet.11ty.js

module.exports = class {
  data() {
    return {
      permalink: '/main.css'
    }
  }

  render() {
    return this.doTheCompileWorkAndGetTheResultingString()
  }

  doTheCompileWorkAndGetTheResultingString() {
    return postcss([...])
  }
}
/* _includes/styles.css */

body {
  font-family: sans-serif;

  /* lives in `_includes` so eleventy automatically watches it for me */
}

@araphiel
Copy link

@araphiel araphiel commented Sep 17, 2019

For anyone stumbling across here, this is my general asset setup for static site generators.

.
├── package.json
├── .eleventy.js
├── site
|     └ assets
|     └ (general page files)
└── src
      └ *.scss
      └ *.js
  • Webpack - (Basic Config)

    • Pre-processed files are saved + edited in the src dir.
    • Webpack outputs asset files to site/assets dir.
  • Eleventy

    • Use addPassthroughCopy to include our post-processed assets site/assets in the build.
    • 11ty ignores anything in .gitignore (aka your webpack assets) so we use setUseGitIgnore(false) to rely on .eleventyignore instead.
/* .eleventy.js */

module.exports = eleventyConfig => {
  eleventyConfig.addPassthroughCopy("site/assets");
  eleventyConfig.setUseGitIgnore(false); 
};
  • NPM Scripts
    • For development, use a node package like npm-run-all to run webpack + 11ty in parallel (both in watch mode)
/* package.json */

  "scripts": {
    "start": "run-s webpack-prod eleventy",
    "dev": "run-p webpack-dev eleventy-dev",
    "eleventy": "eleventy --input=site --output=public",
    "eleventy-dev": "eleventy --input=site --output=public --watch --serve",
    "webpack-prod": "webpack --mode production --env.production",
    "webpack-dev": "webpack --mode development --env.production=false --watch"
  },

@gianpaj
Copy link

@gianpaj gianpaj commented Sep 27, 2019

@jevets do you have a working example of stylesheet.11ty.js with node sass and postcss possibly? 🙏 thanks

@brycewray
Copy link

@brycewray brycewray commented Dec 20, 2019

@jevets I had the same problem with trying to get webpack and Eleventy to "see" each other's stuff. If I used Eleventy's built-in Browsersync instance, textual changes were auto-refreshed but SCSS changes were not. The reverse was true — text yes, SCSS no — if I used webpack-dev-server with Eleventy set only to watch and not to serve.

So, after thinking about the issue for a few days, today I tried a separate instance of Browsersync via browser-sync-webpack-plugin with, again, Eleventy set only to watch and not to serve — and, lo and behold, that worked. Now I get auto-refresh on any change. I have no doubt there are more elegant solutions out there, but this one, finally, did the trick for me.

I reproduced the relevant config files in my now-updated post here, in case they might help someone.

@pascalw
Copy link

@pascalw pascalw commented Dec 20, 2019

What I'm doing which works pretty well is the following:

  1. I run both eleventy serve and webpack --watch in development (using npm-run-all).
  2. Webpack processes assets, directly writes to _site and writes a JSON manifest to src/includes/.webpack/manifest.json. In development this triggers a reload of eleventy whenever I change an asset managed by webpack.
  3. From my templates I can read the webpack manifest (using a template tag) and render the correct stylesheet and JS tags.

This is great because it's fast (webpack runs incremental) and I can utilize webpacks [hash] feature which is great for browser caching.

I have a (slightly outdated) version of this idea up here. I'll update this later with some small improvements I made in my actual (closed source) project.

Only point of friction is that it's inherently a separate system. So adding another entry point is not just a matter of dropping a JS file in a directory, but also needs a change in the webpack config. Not a huge problem since the entrypoints usually don't change much, but still a slight annoyance.

@brycewray
Copy link

@brycewray brycewray commented Dec 21, 2019

@pascalw That smart method raises another thought. I often see people wishing for a pagination function that would give each individual post the “next post” and ”previous post” links (e.g., #529); indeed, I wish for it, too; right now, I manually give each post an idx number (saw that technique on another Eleventy site’s repo). While I see from the repo you linked that you’re using webpack-manifest-plugin to generate the manifest, I wonder how difficult it would be to auto-generate a JSON in _data, akin to how some plugins auto-generate the XML for RSS purposes, that would tie itself to whatever collections one has on a site? In my case, that would be posts. Then it would simply (?) be a matter of pulling from that JSON file and using some JS to call them as appropriate for each page. ...I think. :)

@luwes
Copy link

@luwes luwes commented Dec 22, 2019

can be helpful too

  eleventyConfig.setBrowserSyncConfig({
    files: [
      'public/css',
      'public/js'
    ]
  });

@paulshryock
Copy link
Contributor

@paulshryock paulshryock commented Mar 10, 2020

Updated 1/11/2021

I wanted to share, this is how I'm using Gulp to:

So instead of tasking Eleventy with being a build pipeline that can do everything else, I'm including Eleventy in my external build pipeline, where everything else is handled.

Gulp is invoking Eleventy via the API, instead of Eleventy being invoked via the CLI. This lets me grab the HTML built by Eleventy in a Gulp stream, where I can then post-process the HTML which was built by Eleventy.

From the CLI:

  • gulp develop to build for development
  • gulp serve to build for development, serve everything, and watch for changes
  • gulp build to build for production

@zomars
Copy link

@zomars zomars commented Aug 26, 2020

@jevets I like your solution, would it be possible to do something like that but for bundling JS?

@darthmall
Copy link

@darthmall darthmall commented Nov 3, 2020

I like the idea of keeping asset management independent of 11ty, but I think it would be nice to have some recipes in the docs for stuff like cache busting. It's great to allow people the freedom to configure everything the way they want, but it's also nice to not have to blaze your own trail for every task — especially common tasks like cache busting.

@darthmall
Copy link

@darthmall darthmall commented Nov 14, 2020

I just wrote up a technique for processing styles/scripts using Eleventy that allows you to transform assets (i.e. compile SCSS into CSS, or bundle your JS) and use hashed file names for cache-busting.

I think it's a bit abusive of things like Eleventy's global data files, but it has the advantage of not requiring additional dependencies like Webpack or Parcel. I can't necessarily recommend that you do this, but perhaps it can serve as a strawman for coming up with an official technique for managing assets in Eleventy.

@brycewray
Copy link

@brycewray brycewray commented Nov 14, 2020

I just wrote up a technique for processing styles/scripts using Eleventy that allows you to transform assets (i.e. compile SCSS into CSS, or bundle your JS) and use hashed file names for cache-busting.

I think it's a bit abusive of things like Eleventy's global data files, but it has the advantage of not requiring additional dependencies like Webpack or Parcel. I can't necessarily recommend that you do this, but perhaps it can serve as a strawman for coming up with an official technique for managing assets in Eleventy.

Very nice! I recently posted this article about doing it with a PostCSS plugin, but your solution is better and definitely more inclusive, I think.

@andyford
Copy link

@andyford andyford commented Jan 9, 2021

I know not everyone is using Netlify, but I was trying to find a solution for asset bundling in Eleventy and found that it's a built-in option with Netlify either through the admin uI or the netlify.toml file. Here's a blog post about it: https://www.netlify.com/blog/2019/08/05/control-your-asset-optimization-settings-from-netlify.toml/
...even if you have one single /assets/style.css file linked in the HTML, the bundling feature will turn it into something like https://[SOME_HASH].cloudfront.net/css/[ANOTHER_HASH]/assets/style.css

@SimonEast
Copy link

@SimonEast SimonEast commented May 11, 2021

Adding another vote for this. I realise that people want to use different pipelines, but since SASS and JS bundling is so common, it would be great to have a recommended starting point for newcomers where they can easily drop in to their project and get up and running fast.

I've always found webpack pretty slow and unwieldy, so am hoping Vite.js overtakes it at some point (from the creators of Vue JS). Whether it's mature enough to use as the default, I'm not sure, but it is designed to be pre-configured for most typical tasks, which is the kind of thing that would be great to have in Eleventy.

@zomars
Copy link

@zomars zomars commented May 11, 2021

I found using esbuild it's pretty simple and straightforward:

// /assets/js/main.11ty.js
module.exports = class {
  data() {
    return {
      layout: '',
      permalink: false,
      eleventyExcludeFromCollections: true,
    };
  }

  async render(data) {
    /* https://esbuild.github.io/getting-started/#build-scripts */
    require('esbuild')
      .build({
        entryPoints: ['_includes/js/main.js'],
        bundle: true,
        outfile: '_site/assets/js/main.js',
      })
      .catch(() => process.exit(1));
  }
};

And for CSS:

// /assets/css/style.11ty.js
const path = require('path')
const sass = require('node-sass-promise')
const CleanCSS = require('clean-css')

const inputFile = path.join(__dirname, '../../_includes/scss/main.scss')
const outputFile = path.join(__dirname, '../../assets/css/style.css')

module.exports = class {
  data() {
    return {
      layout: '',
      permalink: 'assets/css/style.css',
      eleventyExcludeFromCollections: true
    }
  }

  async render() {
    const { css } = await sass.render({ file: inputFile })
    const output = new CleanCSS({}).minify(css.toString()).styles

    return output
  }
}

@mandrasch
Copy link

@mandrasch mandrasch commented Apr 8, 2022

Just my two cents: I found nice and interesting examples as well, e.g. https://hipsterbrown.com/musings/musing/esbuild-with-11ty/ or https://github.com/jamshop/eleventy-plugin-esbuild (and so many others as mentioned here). Very exciting to see so many examples!

While this is a very nice challenge for nerds to test on which packages they want to rely - in my personal opinion this could lead to "decision fatigue" and a robust starter would be good for many users of 11ty.

A good and simple test case could be bootstrap5 I guess, because scss and js compiling is needed as well PostCSS autoprefixer is recommended (https://getbootstrap.com/docs/5.0/getting-started/download/#source-files).

npm install bootstrap @popperjs/core --save-dev
/* app.js */
import 'bootstrap';
/* app.scss,
   see: https://getbootstrap.com/docs/5.0/customize/sass/ */
@import "~bootstrap/scss/bootstrap";

I recently found https://github.com/bergwerk/11ty-mix (Laravel Mix => Webpack) for this and got it working by modifying the package.json as suggested for my own projects. As far as I understand https://laravel-mix.com/ was built for the same challenge - keeping asset building simple for devs.

(Here is a simple guide for bootstrap5 and laravel mix in general https://5balloons.info/setting-up-bootstrap-5-workflow-using-laravel-mix-webpack/, laravel mix also has autoprefixer automatically activated https://laravel-mix.com/docs/6.0/autoprefixer)

But the most readable and accessible way for (new) 11ty users would probably be using the 'afterBuild' hook in eleventy config? :-)

@SimonEast
Copy link

@SimonEast SimonEast commented Apr 8, 2022

a robust starter would be good for many users of 11ty ... A good and simple test case could be bootstrap5 I guess ...

Yes, very much agreed. A robust quickstart guide to enabling things like Bootstrap, SCSS and JS would be much appreciated, otherwise it takes a fair bit of research to read through the options of varying quality to work out which is going to work best.

Even Jekyll has better SCSS support out-of-the-box than Eleventy at the moment.

@simplerethink
Copy link

@simplerethink simplerethink commented Apr 8, 2022

@mandrasch @SimonEast - have you tried the Vite plugin and associated youtube video?

@mandrasch
Copy link

@mandrasch mandrasch commented Apr 8, 2022

@mandrasch @SimonEast - have you tried the Vite plugin and associated youtube video?

No, haven't seen or tried. Thanks very much, exciting times ahead! 👍 (Although we'd have to wait for stable release of Eleventy 2.0+ to use this in production if I understand it correctly. So for Eleventy 1.0 a simple starter would be still a nice improvement)

@simplerethink
Copy link

@simplerethink simplerethink commented Apr 8, 2022

@mandrasch I've been building a site on eleventy 2.0 for a few weeks now (not production yet) and haven't come across any issues preventing me from continuing.

@mandrasch
Copy link

@mandrasch mandrasch commented Apr 16, 2022

@mandrasch I've been building a site on eleventy 2.0 for a few weeks now (not production yet) and haven't come across any issues preventing me from continuing.

Thanks for sharing your experience on this!

Optional information for the interested readers regarding Eleventy v1:
Got laravel-mix [webpack / nodejs] working for Bootstrap 5 based on the https://github.com/bergwerk/11ty-mix example. Integrated it into https://github.com/mandrasch/11ty-plain-bs5. 🥳

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement needs-votes
Projects
None yet
Development

No branches or pull requests