Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use Go Modules for Theme Component dependency management #5911

Closed
bep opened this issue Apr 29, 2019 · 43 comments

Comments

@bep
Copy link
Member

commented Apr 29, 2019

This has been mentioned before (I will track down and close that issue), but I'll start fresh.

So, Hugo Themes and Theme Components are very powerful constructs, but the current manual installation/dependency management leaves a lot to be desired. I have tinkered with the idea of using some of the ideas from Go to create a hugo get type of command that could do this stuff, but I imagined lots of work, reinventing the wheel, so to speak.

Today I did some quick tests with Go Modules, and it works surprisingly well with non-Go type of modules. We would need to add some small layer of logic in Hugo to support it, but we would then get really, really good dependency management in Hugo.

To add a new dependency to a theme or a project would then look like this:

  1. go get -u github.com/bep/my-shortcodes
  2. Add github.com/bep/my-shortcodes to the theme slice in config.toml (and yes, @kaushalmodi, we should add an alias for that config key). Note that we of course still will support the /themes/my-shortcodes variant.

When you want to update your dependencies, you do go get -u to get the latest minor or patch release etc.

Note that there can only be one version of my-shortcodes in a Hugo project at a given time, which is how Minimal Version Selection comes into play. We need to investigate github.com/bep/my-shortcodes/v2 vs namespace vs names etc., but that can probably wait.

Note that the above will require that you have Go (>= v1.11) installed, but we should also probably add a hugo mod vendor command that writes all the dependencies to a folder in the project which could be used without needing Go.

Shout if you think this is a terrible idea, @kaushalmodi @onedrawingperday @digitalcraftsman @budparr @regisphilibert @RickCogley, and gang.

@bep bep added the Enhancement label Apr 29, 2019

@bep bep added this to the v0.56 milestone Apr 29, 2019

@bep bep self-assigned this Apr 29, 2019

@budparr

This comment has been minimized.

Copy link

commented Apr 29, 2019

I Love this idea for getting us away from submodules.

If I understand this correctly, everything would be packaged up before our site hits a deployment service/script. It's not clear in my mind exactly: Would we be sending a binary to deploy, so instead of building with Hugo, Netlify—for instance—is building the site with Hugo_site_somethingspecific or whatever?

@kaushalmodi

This comment has been minimized.

Copy link
Member

commented Apr 29, 2019

So this would need the users of themes with "theme components" to have go >= v1.11 installed, right?

Update: Or may be, for the folks with go >= v1.11 installed, they can use the go get approach, others can fall back to using git sub modules?

I am just trying to understand how this will pan out on Netlify.

@regisphilibert

This comment has been minimized.

Copy link

commented Apr 29, 2019

I find it very appealing.
If there's a Go fallback for users using < v1.11, it's great, but the current solution could also suffice as a fallback.

I am just trying to understand how this will pan out on Netlify.

Same here, worst case scenario we have special build command to run for when a dep needs to be updated/added and it's totally fine with me.
Best case: Netlify gets/updates the dep as it builds.

@bep

This comment has been minimized.

Copy link
Member Author

commented Apr 29, 2019

@budparr there is no "binary"; the dependencies (shortcodes, themes etc.) are purely files and directories.

So, Netlify (at least their latest build image) has Go 1.12 installed, so that should not be a problem. And this is kind of analogous to JS projects needing NPM.

We can possibly improve this in the future if Go exports these commands as an API somewhat.

So for Netlify (given that it has Go 1.12 installed) you would not do anything special on the build side.

Running:

hugo

Will pull down any (new) dependency to the Go Module cache (which I assume is or could be easily cached between builds) and build.

For the situations where you want to create "standalone Hugo" distribution, you would run:

hugo mod vendor

The above would write the dependencies to /vendor (or something) below your Hugo project, and it will build fine even on servers without Go installed.

@budparr

This comment has been minimized.

Copy link

commented Apr 29, 2019

Yes, of course. I wasn't thinking straight. Makes perfect sense; I was just trying to work out deployment in my head. I like the stand alone solution as well.

@regisphilibert

This comment has been minimized.

Copy link

commented Apr 29, 2019

Very exciting!

@bep

This comment has been minimized.

Copy link
Member Author

commented Apr 29, 2019

Or may be, for the folks with go >= v1.11 installed, they can use the go get approach, others can fall back to using git sub modules?

Yes, submodules etc. will still work. Also see the hugo mod vendor thing.

@onedrawingperday

This comment has been minimized.

Copy link
Contributor

commented Apr 29, 2019

This looks great. Particularly the planned hugo mod vendor command, since it will make this feature host agnostic.

@larzza

This comment has been minimized.

Copy link

commented Apr 29, 2019

If I understand this correctly, it means that this could be used to download any git repository to use as a dependency, for example Bootstrap (which I today add as a git submodule in the assets directory for sass customizing)?

@bep

This comment has been minimized.

Copy link
Member Author

commented Apr 29, 2019

@larzza this is, in general, true, but there is a caveat: It will be added as a Hugo Theme Component (which has a fixed folder structure: layouts, static, assets ...), so anything outside those will currently have no effect. We could improve on that, but that is outside of the scope of this particular issue.

@bep

This comment has been minimized.

Copy link
Member Author

commented Apr 29, 2019

That said, there are some exploration left to do here -- thinking about it, I'm not totally sure how a submodule inside a dependency would work, etc.

@bep

This comment has been minimized.

Copy link
Member Author

commented Apr 30, 2019

Google just announced that they are hosting a proxy server for these modules which should speed things up, and it is supposed to make multi-modules easier; yet to be tested.

@Jos512

This comment has been minimized.

Copy link

commented May 1, 2019

I'm concerned that this makes it harder for new people to use Hugo. It introduces an additional dependency (Go), which needs to be installed and updated. Plus that can generate its own error messages.

Every so often on the forum we already get people who get stuck with the well-documented quick start. I imagine that having to install Go and run go commands leads to much more confusion and people giving up on Hugo.

@bep

This comment has been minimized.

Copy link
Member Author

commented May 1, 2019

@Jos512 what alternative would you suggest?

@regisphilibert

This comment has been minimized.

Copy link

commented May 1, 2019

I thought you had to install Go to use Hugo in the first place...

If not, couldn't we wrap a default go get into the Hugo CLI? hugo get -u github.com/bep/my-shortcodes

@bep

This comment has been minimized.

Copy link
Member Author

commented May 1, 2019

If not, couldn't we wrap a default go get into the Hugo CLI? hugo get -u github.com/bep/my-shortcodes

Yes, and we will probably do that (to be able to wrap it with the Hugo stuff we need), but you will even then currently need to have Go installed. I have not looked closely into this, but it looks like the Go Modules stuff isn't available as methods/functions from the outside, so you need to interact with the binary. I do, however, suspect that this will change in the future. I think they have kept this API close to their chest until it is carved in stone.

And no, you don't need Go installed to use Hugo. But I would say that it would feel more natural to me to install Go to get package management than having to install Node/NPM which feels more foreign in this context.

@regisphilibert

This comment has been minimized.

Copy link

commented May 1, 2019

But I would say that it would feel more natural to me to install Go to get package management than having to install Node/NPM which feels more foreign in this context.

I agree. It seemed no one ever complained about having to install node in order to build a website.I don't know of any framework which will install node + npm for you.

@larzza

This comment has been minimized.

Copy link

commented May 1, 2019

It seems kind of strange (but pretty cool) to package (a part of) Go in an executable built by Go. It’s also kind of strange to have a dependency on Go from a stand-alone executable, i.e. Hugo. (I think docker would be of good use to bundle things up, then of course the user has to install docker, but not git and hugo and go and....)

Is it possible to elaborate on the usecase? Is a theme supposed to be a ”Go module”? Or just components in a theme... shortcodes, partials etc? Or is the scenario that it will be possible to mix.

If I develop a theme I guess I would like to have it as a git submodule (or just an ordinary git repo) in the themes directory in my hugo site and then add module dependencies to public available shortcodes and partials I find useful.

@bep

This comment has been minimized.

Copy link
Member Author

commented May 1, 2019

Is it possible to elaborate on the usecase?

When I start a new site, I want to do add something like this:

imports:
- github.com/bep/mynicetheme
- github.com/bep/common-shortcodes
- github.com/budparr/hugo-components/useful-partials

And then, when I run:

hugo

All of the dependencies (including any transitive dependencies) is set up for me and cached as effectively as possible. Then I want easy ways to update dependencies on change and many of the other benefits that come with a proper package manager.

Most of the above may be possible using Git submodules, but it isn't practical (your head will get dizzy if you start to think about transitive dependencies) -- which is why Hugo Theme Components are so criminally underused.

@larzza

This comment has been minimized.

Copy link

commented May 1, 2019

Thanks, got it!

@bep bep pinned this issue May 1, 2019

@gcushen

This comment has been minimized.

Copy link

commented May 2, 2019

Can we statically link Go/Go Modules so that there's no need for users to install Go?

@bep

This comment has been minimized.

Copy link
Member Author

commented May 2, 2019

@gcushen what/how?

@bep

This comment has been minimized.

Copy link
Member Author

commented May 3, 2019

I have been doing a quick implementation today. I have some challenges left, but it fits surprisingly well into the existing theme code.

See https://github.com/bep/my-modular-site

For a very lean site ...

A positive side to this that I didn't know before is that you can have submodules in subfolders that can have its own version, which is super useful, see

https://github.com/bep/hugotestmods

@moorereason

This comment has been minimized.

Copy link
Contributor

commented May 3, 2019

@bep, this is an interesting idea.

I'm not super-excited about requiring Go for theme development. I'm inclined to add this as a "preview feature" similar to who Go handled modules initially (check for HUGO056MODULE=on in env).

I agree that we're all hoping for the day when the Go team exports the modules packages. Can we keep the module feature in preview until they do? I wonder how long that will be. Perhaps we should ask Steve what the team is thinking.

@larzza

This comment has been minimized.

Copy link

commented May 3, 2019

Cool!

I'm not super-excited about requiring Go for theme development.

Isn't the idea that the way we do it today will still exist?

A positive side to this that I didn't know before is that you can have submodules in subfolders that can have its own version, which is super useful, see

https://github.com/bep/hugotestmods

About multi module repos:
https://github.com/golang/go/wiki/Modules#faqs--multi-module-repositories

I note that the recommendation is not to use multi module repos, but maybe this is a case of a perfect fit. The possibility to tag a subdirectory in git with its own separate tag was new to me.

@bep

This comment has been minimized.

Copy link
Member Author

commented May 3, 2019

Isn't the idea that the way we do it today will still exist?

Yes.

bep added a commit that referenced this issue Jun 3, 2019

Add Hugo Modules
Fixes #5973
Fixes #5996
Fixes #6010
Fixes #5911
Fixes #5940

bep added a commit that referenced this issue Jun 10, 2019

Add Hugo Modules
Fixes #5973
Fixes #5996
Fixes #6010
Fixes #5911
Fixes #5940

bep added a commit that referenced this issue Jun 10, 2019

Add Hugo Modules
Fixes #5973
Fixes #5996
Fixes #6010
Fixes #5911
Fixes #5940

bep added a commit that referenced this issue Jun 10, 2019

Add Hugo Modules
Fixes #5973
Fixes #5996
Fixes #6010
Fixes #5911
Fixes #5940

bep added a commit that referenced this issue Jun 11, 2019

Add Hugo Modules
Fixes #5973
Fixes #5996
Fixes #6010
Fixes #5911
Fixes #5940

bep added a commit that referenced this issue Jun 16, 2019

Add Hugo Modules
Fixes #5973
Fixes #5996
Fixes #6010
Fixes #5911
Fixes #5940

bep added a commit that referenced this issue Jun 18, 2019

Add Hugo Modules
Fixes #5973
Fixes #5996
Fixes #6010
Fixes #5911
Fixes #5940

bep added a commit that referenced this issue Jun 18, 2019

Add Hugo Modules
Fixes #5973
Fixes #5996
Fixes #6010
Fixes #5911
Fixes #5940

bep added a commit that referenced this issue Jun 20, 2019

Add Hugo Modules
Fixes #5973
Fixes #5996
Fixes #6010
Fixes #5911
Fixes #5940

bep added a commit that referenced this issue Jun 21, 2019

Add Hugo Modules
Fixes #5973
Fixes #5996
Fixes #6010
Fixes #5911
Fixes #5940

bep added a commit that referenced this issue Jun 26, 2019

Add Hugo Modules
Fixes #5973
Fixes #5996
Fixes #6010
Fixes #5911
Fixes #5940

bep added a commit that referenced this issue Jun 26, 2019

Add Hugo Modules
Fixes #5973
Fixes #5996
Fixes #6010
Fixes #5911
Fixes #5940

bep added a commit that referenced this issue Jun 28, 2019

Add Hugo Modules
Fixes #5973
Fixes #5996
Fixes #6010
Fixes #5911
Fixes #5940

bep added a commit that referenced this issue Jul 3, 2019

Add Hugo Modules
Fixes #5973
Fixes #5996
Fixes #6010
Fixes #5911
Fixes #5940

bep added a commit that referenced this issue Jul 5, 2019

Add Hugo Modules
Fixes #5973
Fixes #5996
Fixes #6010
Fixes #5911
Fixes #5940

bep added a commit that referenced this issue Jul 6, 2019

Add Hugo Modules
Fixes #5973
Fixes #5996
Fixes #6010
Fixes #5911
Fixes #5940
Fixes #6074
@bep

This comment has been minimized.

Copy link
Member Author

commented Jul 6, 2019

@larzza @regisphilibert @budparr @RickCogley @digitalcraftsman @onedrawingperday and gang; this branch is now back to a working state and I will wrap it up in a few days. This is a classic example where the first 90% was easy, but the final 10 ... With the introduction of the very flexible disk mounts (now including content) required me to take a step back and redesign parts of the filesystem handling.

The config below is from https://github.com/bep/my-modular-site and shows the syntax.

In short, the goal is a module definition replaces most of the "dir config" in Hugo (configDir, staticDir etc.). I have created adapters to handle the old setup, but this should be much easier to grasp, esp. for bigger projects.

Summary points:

  • Every Hugo Component (main project, shortcode collection etc.) can have a module configuration in their config.toml (or similar).
  • All of these can define a set of mounts (there are defaults if not sets).
  • A mount is:
    • source: Directory path to the source. For the main project, this can be absolute.
    • target: The mount point in the project. This must start with the component folder, e.g. content/blog.
    • lang: Language code, only relevant for content mounts and static mounts when in multihost mode.
  • You can also define mounts when you import a component; if so, any mounts definition in the imported component will be ignored. This can be especially useful when importing non-Hugo-projects (e.g. Bootstrap SCSS) or if you only need a small part of a theme.

All of the above can be used with components inside the old themes folder or as components managed (version, dependency management) by the new hugo mod commands -- which is a whole another cool story.

The above may look technical, but is in my head super duper cool.

[module]
[[module.mounts]]
source="content"
target="content"
[[module.imports]]
path="github.com/bep/hugotestmods/mymounts"
[[module.imports.mounts]]
source="myassets/subfolder"
target="assets/images"
[[module.imports.mounts]]
source="mydata/subfolder"
target="data/datakey"
[[module.imports]]
path="github.com/bep/hugotestmods/mypartials"
[[module.imports]]
path="github.com/bep/hugo-fresh"

bep added a commit that referenced this issue Jul 10, 2019

Add Hugo Modules
Fixes #5973
Fixes #5996
Fixes #6010
Fixes #5911
Fixes #5940
Fixes #6074
Fixes #6082

bep added a commit that referenced this issue Jul 10, 2019

Add Hugo Modules
Fixes #5973
Fixes #5996
Fixes #6010
Fixes #5911
Fixes #5940
Fixes #6074
Fixes #6082

bep added a commit that referenced this issue Jul 17, 2019

Add Hugo Modules
Fixes #5973
Fixes #5996
Fixes #6010
Fixes #5911
Fixes #5940
Fixes #6074
Fixes #6082

bep added a commit that referenced this issue Jul 19, 2019

Add Hugo Modules
Fixes #5973
Fixes #5996
Fixes #6010
Fixes #5911
Fixes #5940
Fixes #6074
Fixes #6082
Fixes #6092

bep added a commit that referenced this issue Jul 21, 2019

Add Hugo Modules
Fixes #5973
Fixes #5996
Fixes #6010
Fixes #5911
Fixes #5940
Fixes #6074
Fixes #6082
Fixes #6092

bep added a commit that referenced this issue Jul 22, 2019

Add Hugo Modules
Fixes #5973
Fixes #5996
Fixes #6010
Fixes #5911
Fixes #5940
Fixes #6074
Fixes #6082
Fixes #6092

bep added a commit that referenced this issue Jul 23, 2019

Add Hugo Modules
Fixes #5973
Fixes #5996
Fixes #6010
Fixes #5911
Fixes #5940
Fixes #6074
Fixes #6082
Fixes #6092

bep added a commit that referenced this issue Jul 24, 2019

Add Hugo Modules
This commit implements Hugo Modules.

This is a broad subject, but some keywords include:

* A new `module` configuration section where you can import almost anything. You can configure both your own file mounts nd the file mounts of the modules you import. This is the new recommended way of configuring what you earlier put in `configDir`, `staticDir` etc. And it also allows you to mount folders in non-Hugo-projects, e.g. the `SCSS` folder in the Bootstrap GitHub project.
* A module consists of a set of mounts to the standard 7 component types in Hugo: `static`, `content`, `layouts`, `data`, `assets`, `i18n`, and `archetypes`. Yes, Theme Components can now include content, which should be very useful, especially in bigger multilingual projects.
* Modules not in your local file cache will be downloaded automatically and even "hot replaced" while the server is running.
* Hugo Modules supports and encourages semver versioned modules, and uses the minimal version selection algorithm to resolve versions.
* A new set of CLI commands are provided to manage all of this: `hugo mod init`,  `hugo mod get`,  `hugo mod graph`,  `hugo mod tidy`, and  `hugo mod vendor`.

All of the above is backed by Go Modules.

Fixes #5973
Fixes #5996
Fixes #6010
Fixes #5911
Fixes #5940
Fixes #6074
Fixes #6082
Fixes #6092

bep added a commit that referenced this issue Jul 24, 2019

Add Hugo Modules
This commit implements Hugo Modules.

This is a broad subject, but some keywords include:

* A new `module` configuration section where you can import almost anything. You can configure both your own file mounts nd the file mounts of the modules you import. This is the new recommended way of configuring what you earlier put in `configDir`, `staticDir` etc. And it also allows you to mount folders in non-Hugo-projects, e.g. the `SCSS` folder in the Bootstrap GitHub project.
* A module consists of a set of mounts to the standard 7 component types in Hugo: `static`, `content`, `layouts`, `data`, `assets`, `i18n`, and `archetypes`. Yes, Theme Components can now include content, which should be very useful, especially in bigger multilingual projects.
* Modules not in your local file cache will be downloaded automatically and even "hot replaced" while the server is running.
* Hugo Modules supports and encourages semver versioned modules, and uses the minimal version selection algorithm to resolve versions.
* A new set of CLI commands are provided to manage all of this: `hugo mod init`,  `hugo mod get`,  `hugo mod graph`,  `hugo mod tidy`, and  `hugo mod vendor`.

All of the above is backed by Go Modules.

Fixes #5973
Fixes #5996
Fixes #6010
Fixes #5911
Fixes #5940
Fixes #6074
Fixes #6082
Fixes #6092

bep added a commit that referenced this issue Jul 24, 2019

Add Hugo Modules
This commit implements Hugo Modules.

This is a broad subject, but some keywords include:

* A new `module` configuration section where you can import almost anything. You can configure both your own file mounts nd the file mounts of the modules you import. This is the new recommended way of configuring what you earlier put in `configDir`, `staticDir` etc. And it also allows you to mount folders in non-Hugo-projects, e.g. the `SCSS` folder in the Bootstrap GitHub project.
* A module consists of a set of mounts to the standard 7 component types in Hugo: `static`, `content`, `layouts`, `data`, `assets`, `i18n`, and `archetypes`. Yes, Theme Components can now include content, which should be very useful, especially in bigger multilingual projects.
* Modules not in your local file cache will be downloaded automatically and even "hot replaced" while the server is running.
* Hugo Modules supports and encourages semver versioned modules, and uses the minimal version selection algorithm to resolve versions.
* A new set of CLI commands are provided to manage all of this: `hugo mod init`,  `hugo mod get`,  `hugo mod graph`,  `hugo mod tidy`, and  `hugo mod vendor`.

All of the above is backed by Go Modules.

Fixes #5973
Fixes #5996
Fixes #6010
Fixes #5911
Fixes #5940
Fixes #6074
Fixes #6082
Fixes #6092

bep added a commit that referenced this issue Jul 24, 2019

Add Hugo Modules
This commit implements Hugo Modules.

This is a broad subject, but some keywords include:

* A new `module` configuration section where you can import almost anything. You can configure both your own file mounts nd the file mounts of the modules you import. This is the new recommended way of configuring what you earlier put in `configDir`, `staticDir` etc. And it also allows you to mount folders in non-Hugo-projects, e.g. the `SCSS` folder in the Bootstrap GitHub project.
* A module consists of a set of mounts to the standard 7 component types in Hugo: `static`, `content`, `layouts`, `data`, `assets`, `i18n`, and `archetypes`. Yes, Theme Components can now include content, which should be very useful, especially in bigger multilingual projects.
* Modules not in your local file cache will be downloaded automatically and even "hot replaced" while the server is running.
* Hugo Modules supports and encourages semver versioned modules, and uses the minimal version selection algorithm to resolve versions.
* A new set of CLI commands are provided to manage all of this: `hugo mod init`,  `hugo mod get`,  `hugo mod graph`,  `hugo mod tidy`, and  `hugo mod vendor`.

All of the above is backed by Go Modules.

Fixes #5973
Fixes #5996
Fixes #6010
Fixes #5911
Fixes #5940
Fixes #6074
Fixes #6082
Fixes #6092

@bep bep closed this in #5960 Jul 24, 2019

@bep bep unpinned this issue Aug 13, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
10 participants
You can’t perform that action at this time.