New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposal: Add support for pipeline-based content processing #3207

Closed
digitalcraftsman opened this Issue Mar 21, 2017 · 15 comments

Comments

Projects
None yet
@digitalcraftsman
Member

digitalcraftsman commented Mar 21, 2017

@spf13 proposed in his roadmap for Hugo (see #3205) the introduction of pipelines to process content files, assets and other media. Based on predefined triggers the pipelined media get processed by a command associated with the triggers. Pipelines feature a queue that buffers incoming media until they're ready to be processed.

Potential usecases are:

  • image cropping, resizing and compression via a special filename format
  • compiling stylesheets and JS (i.e. Less, Sass and CoffeeScript)
  • minification and concatenation of stylesheets and JS

Such applications are likely to require external tools and could lay the foundation for plugins (which will not be focused in this issue) that can be registered as a command for a trigger.

Related issues:

Note: this issue just outlines the concept around pipelines and potential applications. A more formal specification has still to be defined. Leave a comment with your ideas to shape this draft further.

@bogem

This comment has been minimized.

Show comment
Hide comment
@bogem

bogem Mar 21, 2017

Contributor

Is it out of responsibility of Hugo? Theme developers can minify their themes, if they want. But built-in HTML minification and image compression would be really great and it also isn't hard to implement.
I also can't imagine, how users can configure pipelines.

Contributor

bogem commented Mar 21, 2017

Is it out of responsibility of Hugo? Theme developers can minify their themes, if they want. But built-in HTML minification and image compression would be really great and it also isn't hard to implement.
I also can't imagine, how users can configure pipelines.

@tech4him1

This comment has been minimized.

Show comment
Hide comment
@tech4him1

tech4him1 Mar 21, 2017

Pipelines might add a lot more complexity to Hugo, versus just something like "pre" and "post" hooks to allow Hugo to hook into other workflows more easily.

tech4him1 commented Mar 21, 2017

Pipelines might add a lot more complexity to Hugo, versus just something like "pre" and "post" hooks to allow Hugo to hook into other workflows more easily.

@rdwatters

This comment has been minimized.

Show comment
Hide comment
@rdwatters

rdwatters Mar 22, 2017

Contributor

@digitalcraftsman Maybe we can get some more feedback if you expand a bit on the differences between, say, a pipeline, a plugin, and an external helper?

Contributor

rdwatters commented Mar 22, 2017

@digitalcraftsman Maybe we can get some more feedback if you expand a bit on the differences between, say, a pipeline, a plugin, and an external helper?

@rdwatters

This comment has been minimized.

Show comment
Hide comment
@rdwatters

rdwatters Mar 22, 2017

Contributor

Ah, finally found it. My thoughts about Hugo's purpose and scope is basically the same as when I first started using the project 16 months ago:

https://discuss.gohugo.io/t/roadmap-to-hugo-v1-0/2278/2?u=rdwatters

In other words, I agree with @bogem. The best thing that Hugo can do is focus on content creation and content management, which already seems to be the direction that things are heading:

  1. Write to markup, JSON, ical, etc (i.e., multiple output formats).
  2. Keep things fast, or better, faster (many thanks and kudos to @bep and all the awesome devs for making an SSG that was recently described as "Hugo is so fast it builds my site faster than my browser can reload it")
  3. Don't necessarily couple a CMS layer to Hugo but start laying the infrastructure for other's to create multiple drop-in SPA CMSs (seems like this is already happening with Netlify, Forestry, et al).
  4. Terser, easier template functions for the most common use cases (this is already improved so much since I first started using the tool).
  5. Similar to (3), focus more on how to make the framework easier for theme developers and avoid painting ourselves into a corner with a rigid Wordpress-like everything-is-a-post-or-page data model.
  6. Focus on other content-related pieces that are part and parcel of iterative content management: e.g., image cropping/resizing/optimizing/, link checking/management, etc.

I think the JS/NPM community already has all CSS/SCSS/concat/minification covered in spades. That said, if you think there is a way to bridge the two without overburdening Hugo, that's awesome and I defer to you.

All said, I think the coolest piece from @spf13 's presentation is the content bundling piece, so I'm excited to see what the community says about that as well. Thanks, @digitalcraftsman !

Contributor

rdwatters commented Mar 22, 2017

Ah, finally found it. My thoughts about Hugo's purpose and scope is basically the same as when I first started using the project 16 months ago:

https://discuss.gohugo.io/t/roadmap-to-hugo-v1-0/2278/2?u=rdwatters

In other words, I agree with @bogem. The best thing that Hugo can do is focus on content creation and content management, which already seems to be the direction that things are heading:

  1. Write to markup, JSON, ical, etc (i.e., multiple output formats).
  2. Keep things fast, or better, faster (many thanks and kudos to @bep and all the awesome devs for making an SSG that was recently described as "Hugo is so fast it builds my site faster than my browser can reload it")
  3. Don't necessarily couple a CMS layer to Hugo but start laying the infrastructure for other's to create multiple drop-in SPA CMSs (seems like this is already happening with Netlify, Forestry, et al).
  4. Terser, easier template functions for the most common use cases (this is already improved so much since I first started using the tool).
  5. Similar to (3), focus more on how to make the framework easier for theme developers and avoid painting ourselves into a corner with a rigid Wordpress-like everything-is-a-post-or-page data model.
  6. Focus on other content-related pieces that are part and parcel of iterative content management: e.g., image cropping/resizing/optimizing/, link checking/management, etc.

I think the JS/NPM community already has all CSS/SCSS/concat/minification covered in spades. That said, if you think there is a way to bridge the two without overburdening Hugo, that's awesome and I defer to you.

All said, I think the coolest piece from @spf13 's presentation is the content bundling piece, so I'm excited to see what the community says about that as well. Thanks, @digitalcraftsman !

@Jos512

This comment has been minimized.

Show comment
Hide comment
@Jos512

Jos512 Apr 1, 2017

I think the JS/NPM community already has all CSS/SCSS/concat/minification covered in spades. That said, if you think there is a way to bridge the two without overburdening Hugo, that's awesome and I defer to you.

That's true @rdwatters . I also doubt how easy this will be; a minifier like html-minifier already has 30+ configurable options. We might not need all of those options in the Hugo use case, but likely still need more than expected. That gives additional code and docs to maintain, tests to develop and possible conflicts to resolve, as well as forum support queries to attend to.

Perhaps content processing by Hugo natively is something for when the Hugo team is bigger?

Jos512 commented Apr 1, 2017

I think the JS/NPM community already has all CSS/SCSS/concat/minification covered in spades. That said, if you think there is a way to bridge the two without overburdening Hugo, that's awesome and I defer to you.

That's true @rdwatters . I also doubt how easy this will be; a minifier like html-minifier already has 30+ configurable options. We might not need all of those options in the Hugo use case, but likely still need more than expected. That gives additional code and docs to maintain, tests to develop and possible conflicts to resolve, as well as forum support queries to attend to.

Perhaps content processing by Hugo natively is something for when the Hugo team is bigger?

@xakz

This comment has been minimized.

Show comment
Hide comment
@xakz

xakz Apr 26, 2017

Pipelined content is a really cool idea but it's probably too much here. Just a pre and post processing hook that integrates with LiveReload is enough.

If Hugo can fetch additionnal parameters from a preprocessing hook to use it in the templates, it's perfect for me. Maybe by parsing the hook stdout as toml/yaml/json. And if hooks are executed by lexical order, it's like a pipelined processing, but out of Hugo itself.

This can be done right now, using a simple Makefile/Gruntfile/Gulpfile/Rakefile/AnythingYouWant but that do not integrate with LiveReload and I'm sure it is what we want.

xakz commented Apr 26, 2017

Pipelined content is a really cool idea but it's probably too much here. Just a pre and post processing hook that integrates with LiveReload is enough.

If Hugo can fetch additionnal parameters from a preprocessing hook to use it in the templates, it's perfect for me. Maybe by parsing the hook stdout as toml/yaml/json. And if hooks are executed by lexical order, it's like a pipelined processing, but out of Hugo itself.

This can be done right now, using a simple Makefile/Gruntfile/Gulpfile/Rakefile/AnythingYouWant but that do not integrate with LiveReload and I'm sure it is what we want.

@bep bep referenced this issue Jun 27, 2017

Closed

Add Content Bags #3651

@andyfleming

This comment has been minimized.

Show comment
Hide comment
@andyfleming

andyfleming Sep 22, 2017

I mostly agree that taking on the whole world of npm, gulp, webpack, javascript, css, and all that might be unnecessary.

However, in first using Hugo, I felt like there wasn't a clear workflow or defaults for where to put javascript and css, especially when starting a new site/theme. It quickly becomes obvious that there are some flexible options, but it seems like it should have a more obvious starting point.

I think it is the responsibility of hugo to have clear defaults for where CSS and JS end up (though it should maintain the flexibility to do whatever you want), but it doesn't need to handle extra pre-processing. Once there are more clear defaults, it may actually unify how others approach layering additional tools on top of hugo.

andyfleming commented Sep 22, 2017

I mostly agree that taking on the whole world of npm, gulp, webpack, javascript, css, and all that might be unnecessary.

However, in first using Hugo, I felt like there wasn't a clear workflow or defaults for where to put javascript and css, especially when starting a new site/theme. It quickly becomes obvious that there are some flexible options, but it seems like it should have a more obvious starting point.

I think it is the responsibility of hugo to have clear defaults for where CSS and JS end up (though it should maintain the flexibility to do whatever you want), but it doesn't need to handle extra pre-processing. Once there are more clear defaults, it may actually unify how others approach layering additional tools on top of hugo.

@tdewolff

This comment has been minimized.

Show comment
Hide comment
@tdewolff

tdewolff Oct 16, 2017

@rdwatters Not to brag, but I think my minify library would not overburden Hugo, it is also build with high performance in mind and typical small pages would minify in under 1 ms. It is possible to use NPM tooling for this, which are surely more battle-tested, but they are usually two orders of magnitude slower. I'd be willing to help integrate it if needed.

EDIT: I just noticed that there is a PR already at #2664

tdewolff commented Oct 16, 2017

@rdwatters Not to brag, but I think my minify library would not overburden Hugo, it is also build with high performance in mind and typical small pages would minify in under 1 ms. It is possible to use NPM tooling for this, which are surely more battle-tested, but they are usually two orders of magnitude slower. I'd be willing to help integrate it if needed.

EDIT: I just noticed that there is a PR already at #2664

@galopin

This comment has been minimized.

Show comment
Hide comment
@galopin

galopin Oct 24, 2017

@digitalcraftsman @rdwatters Everyone should be compressing their images efficiently. There are many third-party solutions out there that do offer comprehensive automated image optimization solutions. They can work out-of-the box with SSGs, especially Cloudinary.

We should all be automating our image compression.
-- Essential Image Optimization, Addy Osmani

galopin commented Oct 24, 2017

@digitalcraftsman @rdwatters Everyone should be compressing their images efficiently. There are many third-party solutions out there that do offer comprehensive automated image optimization solutions. They can work out-of-the box with SSGs, especially Cloudinary.

We should all be automating our image compression.
-- Essential Image Optimization, Addy Osmani

@bep

This comment has been minimized.

Show comment
Hide comment
@bep

bep Dec 15, 2017

Member

I have spent considerable amount of time thinking about "page bundles" (see #3651), which also includes basic image processing (scaling, resizing etc.) -- and in my head the "assets pipeline" model only works for global assets (examples would be the main stylesheet, all PNG images etc.).

For page or template local assets, I find a reactive/lazy/on-demand model easier to reason about and to get up to the scale and performance Hugo users expect.

I have gotten some gray hair wrapping my head around how to handle changes in these bundles. But now, watching the browser reload with updated thumbnails after an edit of the original in Photoshop or similar after I change the thumbnail settings in the template, I can safely say this: That would not have been possible at this speed without some kind of controlled content dependency graph.

Member

bep commented Dec 15, 2017

I have spent considerable amount of time thinking about "page bundles" (see #3651), which also includes basic image processing (scaling, resizing etc.) -- and in my head the "assets pipeline" model only works for global assets (examples would be the main stylesheet, all PNG images etc.).

For page or template local assets, I find a reactive/lazy/on-demand model easier to reason about and to get up to the scale and performance Hugo users expect.

I have gotten some gray hair wrapping my head around how to handle changes in these bundles. But now, watching the browser reload with updated thumbnails after an edit of the original in Photoshop or similar after I change the thumbnail settings in the template, I can safely say this: That would not have been possible at this speed without some kind of controlled content dependency graph.

@regisphilibert

This comment has been minimized.

Show comment
Hide comment
@regisphilibert

regisphilibert Feb 15, 2018

Now that .Resources will be available to any kind of file. (#4381)
Wouldn't it make sense to rely on them for this content processing feature?

{{ $script := .Site.Resources.GetMatch "js/app.js" }}
<script type="text/javascript" src="{{ $script.Minify }}?{{ $style.FingerPrint }}"></script>

Would make sense.

PS: On some very public repos, you'd rather have your minified files committed so your users don't have to npm install/grunt when they start using your theme or upgrading it. This means merging conflict hell! Having hugo do at least the minification would solve that!

regisphilibert commented Feb 15, 2018

Now that .Resources will be available to any kind of file. (#4381)
Wouldn't it make sense to rely on them for this content processing feature?

{{ $script := .Site.Resources.GetMatch "js/app.js" }}
<script type="text/javascript" src="{{ $script.Minify }}?{{ $style.FingerPrint }}"></script>

Would make sense.

PS: On some very public repos, you'd rather have your minified files committed so your users don't have to npm install/grunt when they start using your theme or upgrading it. This means merging conflict hell! Having hugo do at least the minification would solve that!

@stale

This comment has been minimized.

Show comment
Hide comment
@stale

stale bot Jun 15, 2018

This issue has been automatically marked as stale because it has not had recent activity. The resources of the Hugo team are limited, and so we are asking for your help.
If this is a bug and you can still reproduce this error on the master branch, please reply with all of the information you have about it in order to keep the issue open.
If this is a feature request, and you feel that it is still relevant and valuable, please tell us why.
This issue will automatically be closed in the near future if no further activity occurs. Thank you for all your contributions.

stale bot commented Jun 15, 2018

This issue has been automatically marked as stale because it has not had recent activity. The resources of the Hugo team are limited, and so we are asking for your help.
If this is a bug and you can still reproduce this error on the master branch, please reply with all of the information you have about it in order to keep the issue open.
If this is a feature request, and you feel that it is still relevant and valuable, please tell us why.
This issue will automatically be closed in the near future if no further activity occurs. Thank you for all your contributions.

@stale stale bot added the Stale label Jun 15, 2018

@Jos512

This comment has been minimized.

Show comment
Hide comment
@Jos512

Jos512 Jun 16, 2018

I don't think this issue should be closed, mr. Stale Bot. 🙂

I still love to have HTML, JavaScript, and CSS minification + finger printing in Hugo. But not just me, I think the Hugo community will also benefit from it. Particular people that come from platforms like WordPress, and don't know Gulp/Grunt or any other asset processing tool. For them Hugo will become easier to use when that kind of minification is provided out of the box.

And with the SASS support coming in the future, this might be something that's easier to implement.

Jos512 commented Jun 16, 2018

I don't think this issue should be closed, mr. Stale Bot. 🙂

I still love to have HTML, JavaScript, and CSS minification + finger printing in Hugo. But not just me, I think the Hugo community will also benefit from it. Particular people that come from platforms like WordPress, and don't know Gulp/Grunt or any other asset processing tool. For them Hugo will become easier to use when that kind of minification is provided out of the box.

And with the SASS support coming in the future, this might be something that's easier to implement.

@stale stale bot removed the Stale label Jun 16, 2018

@hanzei

This comment has been minimized.

Show comment
Hide comment
@hanzei

hanzei Jul 21, 2018

I think hugo 0.43 and hugo piper added everythink needed for this issue. See http://gohugo.io/hugo-pipes/minification/

hanzei commented Jul 21, 2018

I think hugo 0.43 and hugo piper added everythink needed for this issue. See http://gohugo.io/hugo-pipes/minification/

@bep bep closed this Jul 21, 2018

@bep

This comment has been minimized.

Show comment
Hide comment
@bep

bep Jul 21, 2018

Member

@hanzei really appreciate the work you do on issue tracking. I guess I got a little burned out when I finished implementing all the asset handling stuff.

Member

bep commented Jul 21, 2018

@hanzei really appreciate the work you do on issue tracking. I guess I got a little burned out when I finished implementing all the asset handling stuff.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment