Add exec shortcode #796

Open
bep opened this Issue Jan 14, 2015 · 53 comments

Comments

Projects
None yet
@bep
Member

bep commented Jan 14, 2015

Idea from this post:
http://discuss.gohugo.io/t/inline-ditaa-dot-or-plantuml-sources-in-hugo-posts/589/3

Similar to Highlight. but for any binary that writes to stdout.

Two arguments:

  1. Binary
  2. Args
  • the inner content to stdin

I don't see any obvious issues why we could/should not add this? Security issues? It's up to the developer to use or abuse this?

@bep bep added the Enhancement label Jan 14, 2015

@bep

This comment has been minimized.

Show comment
Hide comment
@bep

bep Jan 15, 2015

Member

Come to think of it, it will give theme-authors the opening to do harmful stuff ... like "rm -r /".

But it's an interesting thought - if we could restrict it to a set of commands.

Just tested cat combined with highlight and some go files. Very useful.

{{< highlight go >}}
{{< exec "cat" "/home/bep/dev/go/src/github.com/spf13/hugo/hugolib/shortcodeparser.go" >}}
{{< / highlight >}}

Note, the above will not make any sense without the fix for #797 - but it's kind of cool.

Member

bep commented Jan 15, 2015

Come to think of it, it will give theme-authors the opening to do harmful stuff ... like "rm -r /".

But it's an interesting thought - if we could restrict it to a set of commands.

Just tested cat combined with highlight and some go files. Very useful.

{{< highlight go >}}
{{< exec "cat" "/home/bep/dev/go/src/github.com/spf13/hugo/hugolib/shortcodeparser.go" >}}
{{< / highlight >}}

Note, the above will not make any sense without the fix for #797 - but it's kind of cool.

@halostatue

This comment has been minimized.

Show comment
Hide comment
@halostatue

halostatue Jan 15, 2015

Contributor

I don’t think it gives theme authors anything—shortcodes aren’t rendered by templates, only content. Right?

Contributor

halostatue commented Jan 15, 2015

I don’t think it gives theme authors anything—shortcodes aren’t rendered by templates, only content. Right?

@anthonyfok

This comment has been minimized.

Show comment
Hide comment
@anthonyfok

anthonyfok Jan 15, 2015

Member

Interesting, and very useful indeed! :-)

Though, for security reasons, I think it should be a configurable option that is turned off by default.
It may also be helpful to print out the exec command for each website rebuild so that the end user is fully aware of what is going on. And yes, restriction to a set of commands (and other needed sanitization) would be a good thing.

Member

anthonyfok commented Jan 15, 2015

Interesting, and very useful indeed! :-)

Though, for security reasons, I think it should be a configurable option that is turned off by default.
It may also be helpful to print out the exec command for each website rebuild so that the end user is fully aware of what is going on. And yes, restriction to a set of commands (and other needed sanitization) would be a good thing.

@bep

This comment has been minimized.

Show comment
Hide comment
@bep

bep Jan 15, 2015

Member

@halostatue no, but shortcodes can be provided by themes. So a shortcode from a theme named coolinnocentshortcode is tempting to use.

But I think this is too useful turn down without some thinking.

EDIT IN: I may be wrong -- if we restrict this to shortcodes only, this will only be available to content files ...

Member

bep commented Jan 15, 2015

@halostatue no, but shortcodes can be provided by themes. So a shortcode from a theme named coolinnocentshortcode is tempting to use.

But I think this is too useful turn down without some thinking.

EDIT IN: I may be wrong -- if we restrict this to shortcodes only, this will only be available to content files ...

@bep

This comment has been minimized.

Show comment
Hide comment
@bep

bep Jan 15, 2015

Member

But it would be cool if we could make some of the obvious available as both template function and shortcode (if they exist on the computer):

  • cat
  • tail
  • curl
  • ??
Member

bep commented Jan 15, 2015

But it would be cool if we could make some of the obvious available as both template function and shortcode (if they exist on the computer):

  • cat
  • tail
  • curl
  • ??
@spf13

This comment has been minimized.

Show comment
Hide comment
@spf13

spf13 Jan 16, 2015

Contributor

Sorry I'm a bit late to this. When I saw this I immediately thought that it was a security issue as you pointed out in your follow up.

I agree that the best approach would be to define a few commonly used commands.

cat & grep alone would be super powerful. Curl may be but could also pose a security risk.

Another concern is that these won't work for Windows.

Contributor

spf13 commented Jan 16, 2015

Sorry I'm a bit late to this. When I saw this I immediately thought that it was a security issue as you pointed out in your follow up.

I agree that the best approach would be to define a few commonly used commands.

cat & grep alone would be super powerful. Curl may be but could also pose a security risk.

Another concern is that these won't work for Windows.

@bep

This comment has been minimized.

Show comment
Hide comment
@bep

bep Jan 16, 2015

Member

Yea, thinking a little about this. I foresee a support nightmare. "I get 'broken pipe' with Photoshop".

I might maintain a private patch for myself, but for Hugo it's maybe better to find a Go implementation similar to cat etc.

My current use case is what Martin Fowler calls "live code" in documentation - but much simpler than his setup, i. e. text extracts from source code on disk (I know about Gists).

Member

bep commented Jan 16, 2015

Yea, thinking a little about this. I foresee a support nightmare. "I get 'broken pipe' with Photoshop".

I might maintain a private patch for myself, but for Hugo it's maybe better to find a Go implementation similar to cat etc.

My current use case is what Martin Fowler calls "live code" in documentation - but much simpler than his setup, i. e. text extracts from source code on disk (I know about Gists).

@rodlogic

This comment has been minimized.

Show comment
Hide comment
@rodlogic

rodlogic Jan 17, 2015

My main motivation for this is so I can plug in tools such as plantuml, ditaa or graphviz to generate diagrams while building the site (Jekyll supports this quite well). AFAIK, this is not possible in Hugo at this point and that a big minus to me.

Although having curl, cat etc as standard shortcodes, it would not address what I am looking for. Either there is a generic exec short code or a specific one for these tools above (here a plugin approach would be best specially since any support questions can be deferred to the developer of the plugin).

My main motivation for this is so I can plug in tools such as plantuml, ditaa or graphviz to generate diagrams while building the site (Jekyll supports this quite well). AFAIK, this is not possible in Hugo at this point and that a big minus to me.

Although having curl, cat etc as standard shortcodes, it would not address what I am looking for. Either there is a generic exec short code or a specific one for these tools above (here a plugin approach would be best specially since any support questions can be deferred to the developer of the plugin).

@bep

This comment has been minimized.

Show comment
Hide comment
@bep

bep Jan 17, 2015

Member

Go's static linking is mostly a blessing, but it also kicks the feet under the plugin word.

Go has a brilliant and simple exec-package that is perfect for this purpose, but I don't see Hugo pulling in plantuml, ditaa ... as specific shortcodes/functions like Pygments.

Maybe if a general Exec could be secured (whitelist?)

Member

bep commented Jan 17, 2015

Go's static linking is mostly a blessing, but it also kicks the feet under the plugin word.

Go has a brilliant and simple exec-package that is perfect for this purpose, but I don't see Hugo pulling in plantuml, ditaa ... as specific shortcodes/functions like Pygments.

Maybe if a general Exec could be secured (whitelist?)

@lalloni

This comment has been minimized.

Show comment
Hide comment
@lalloni

lalloni Jan 25, 2015

I've just found this PR while searching for some way of using plantuml from hugo and I must say that this looks almost exactly like what we need. Thanks!

WRT a possible plugin infrastructure in hugo/go, how about a plugin architecture in which plugins are standalone programs which expect arguments from stdin in some standard form (json, protobuf, etc.) and write answers to stdout using the same format (with possible secondary effects in the output fs directory), this plugins should be registered at the project level (not from themes) and invoked from content or templates.

lalloni commented Jan 25, 2015

I've just found this PR while searching for some way of using plantuml from hugo and I must say that this looks almost exactly like what we need. Thanks!

WRT a possible plugin infrastructure in hugo/go, how about a plugin architecture in which plugins are standalone programs which expect arguments from stdin in some standard form (json, protobuf, etc.) and write answers to stdout using the same format (with possible secondary effects in the output fs directory), this plugins should be registered at the project level (not from themes) and invoked from content or templates.

@bep

This comment has been minimized.

Show comment
Hide comment
@bep

bep Jan 25, 2015

Member

@plalloni well, the "plugin architecure" you want is basically a whitelist for the "exec template function", but who approves the whitelist?

There are great powers in Brad F. something's exec package, but as stated, there may be concerns open it up, even to a static site gen.

Member

bep commented Jan 25, 2015

@plalloni well, the "plugin architecure" you want is basically a whitelist for the "exec template function", but who approves the whitelist?

There are great powers in Brad F. something's exec package, but as stated, there may be concerns open it up, even to a static site gen.

@lalloni

This comment has been minimized.

Show comment
Hide comment
@lalloni

lalloni Jan 25, 2015

@bjornerik Why is a whitelist needed if plugins are not to be registered by theme creators but only by site authors (i.e. a theme could require a certain plugin installed but can not install it by itself, so the site/project owner would be required to install/register the plugin).

lalloni commented Jan 25, 2015

@bjornerik Why is a whitelist needed if plugins are not to be registered by theme creators but only by site authors (i.e. a theme could require a certain plugin installed but can not install it by itself, so the site/project owner would be required to install/register the plugin).

@spf13

This comment has been minimized.

Show comment
Hide comment
@spf13

spf13 Jan 25, 2015

Contributor

@plalloni There aren't "plugins". These would be methods for the template system (which would call exec) and there's not currently any way of restricting them from some templates and not others. Consequently we would need to keep it clean everywhere.

Contributor

spf13 commented Jan 25, 2015

@plalloni There aren't "plugins". These would be methods for the template system (which would call exec) and there's not currently any way of restricting them from some templates and not others. Consequently we would need to keep it clean everywhere.

@bep bep added the discussion label Jan 27, 2015

@lalloni

This comment has been minimized.

Show comment
Hide comment
@lalloni

lalloni Jan 27, 2015

@bjornerik Any chance I can get my hands on your patch?

lalloni commented Jan 27, 2015

@bjornerik Any chance I can get my hands on your patch?

@bep

This comment has been minimized.

Show comment
Hide comment
@bep

bep Jan 27, 2015

Member

@plalloni it's not in a shareable state.

Member

bep commented Jan 27, 2015

@plalloni it's not in a shareable state.

@aliafshar

This comment has been minimized.

Show comment
Hide comment
@aliafshar

aliafshar Jan 27, 2015

Limited range of commands won't work for me. This would be a great hook for running arbitrary external commands to perform transforms on part of the content. I will write a patch for my own use.

Limited range of commands won't work for me. This would be a great hook for running arbitrary external commands to perform transforms on part of the content. I will write a patch for my own use.

@halostatue

This comment has been minimized.

Show comment
Hide comment
@halostatue

halostatue Jan 27, 2015

Contributor

I think that this can still be written in a way as to be safe, using front matter or config.*. If I do:

exec.whitelist = [
  "cat",
  "tac",
  "curl"
]

This is something that the user must configure, and any command where the command does not exactly match a string in the whitelist will fail to run.

Contributor

halostatue commented Jan 27, 2015

I think that this can still be written in a way as to be safe, using front matter or config.*. If I do:

exec.whitelist = [
  "cat",
  "tac",
  "curl"
]

This is something that the user must configure, and any command where the command does not exactly match a string in the whitelist will fail to run.

@aliafshar

This comment has been minimized.

Show comment
Hide comment
@aliafshar

aliafshar Jan 27, 2015

Here is my patch #847

Nothing fancy, adds an exec function that will be usable in shortcodes I hope. Usable for my purpose, but needs work for any thoughts of inclusion into the main tree.

Here is my patch #847

Nothing fancy, adds an exec function that will be usable in shortcodes I hope. Usable for my purpose, but needs work for any thoughts of inclusion into the main tree.

@lalloni

This comment has been minimized.

Show comment
Hide comment
@lalloni

lalloni Jan 28, 2015

@aliafshar Thank you! I'll try it ASAP.

lalloni commented Jan 28, 2015

@aliafshar Thank you! I'll try it ASAP.

@dcorb

This comment has been minimized.

Show comment
Hide comment
@dcorb

dcorb Feb 5, 2015

I would love to see this feature in Hugo.
This opens the doors to all kind of external utilities. From basic 'cat' to generating charts with d3js. Something like this:
http://www.scottlogic.com/blog/2014/09/15/jekyll-d3js.html

Security is here the biggest concern, but this would be so useful.

As a follow up: Would be possible to cache "exec" outputs, to prevent slowing down the system?

dcorb commented Feb 5, 2015

I would love to see this feature in Hugo.
This opens the doors to all kind of external utilities. From basic 'cat' to generating charts with d3js. Something like this:
http://www.scottlogic.com/blog/2014/09/15/jekyll-d3js.html

Security is here the biggest concern, but this would be so useful.

As a follow up: Would be possible to cache "exec" outputs, to prevent slowing down the system?

@lalloni

This comment has been minimized.

Show comment
Hide comment
@lalloni

lalloni Feb 6, 2015

WRT security: what if allowed executables must be in some specific site level path like /bin (where can't be installed by theme creators)?

Site owner can symlink from bin to anywhere in the system, and/or install specific programs or scripts for calling tools like plantuml o d3.

lalloni commented Feb 6, 2015

WRT security: what if allowed executables must be in some specific site level path like /bin (where can't be installed by theme creators)?

Site owner can symlink from bin to anywhere in the system, and/or install specific programs or scripts for calling tools like plantuml o d3.

@anthonyfok

This comment has been minimized.

Show comment
Hide comment
@anthonyfok

anthonyfok Feb 6, 2015

Member

Or, how about this:

Normally, when hugo is run with no special flags, the exec statements are printed but not executed;
The exec statements are only run when the option --unsafe is added?

For added safety measure, even with --unsafe, perhaps Hugo would pause and list all exec statements, asking the user whether they are sure if they want to proceed (y/N)? And perhaps an added flag --yes-i-really-know-what-i-am-doing is needed to skip that question?

Member

anthonyfok commented Feb 6, 2015

Or, how about this:

Normally, when hugo is run with no special flags, the exec statements are printed but not executed;
The exec statements are only run when the option --unsafe is added?

For added safety measure, even with --unsafe, perhaps Hugo would pause and list all exec statements, asking the user whether they are sure if they want to proceed (y/N)? And perhaps an added flag --yes-i-really-know-what-i-am-doing is needed to skip that question?

@aliafshar

This comment has been minimized.

Show comment
Hide comment
@aliafshar

aliafshar Feb 9, 2015

And, I changed my mind here. I am running an independent preprocessor on the content. I think the main problem here is that hugo will at some point need a way of being extended with plugins that execute code, not just template shortcodes - makes long-term sense for a system like this.

Then security can be addressed by explicitly allowing plugins. I appreciate Golang is going to make this very hard.

And, I changed my mind here. I am running an independent preprocessor on the content. I think the main problem here is that hugo will at some point need a way of being extended with plugins that execute code, not just template shortcodes - makes long-term sense for a system like this.

Then security can be addressed by explicitly allowing plugins. I appreciate Golang is going to make this very hard.

@lalloni

This comment has been minimized.

Show comment
Hide comment
@lalloni

lalloni Feb 9, 2015

@aliafshar I agree. I've been thinking about "plugins" as executable files/links somewhere in the site directory structure. Hugo could run them as subprocesses and comunicate with them using stdin/out. Could be one-off subprocesses or "server style" processes (for keeping latency as low as possible) attending many requests.

lalloni commented Feb 9, 2015

@aliafshar I agree. I've been thinking about "plugins" as executable files/links somewhere in the site directory structure. Hugo could run them as subprocesses and comunicate with them using stdin/out. Could be one-off subprocesses or "server style" processes (for keeping latency as low as possible) attending many requests.

@timesking

This comment has been minimized.

Show comment
Hide comment

How about this plugin framework, https://github.com/dullgiulio/pingo

@spf13

This comment has been minimized.

Show comment
Hide comment
@spf13

spf13 Apr 28, 2015

Contributor

@timesking It's brand new and will require some testing. I think it has a lot of potential.

Contributor

spf13 commented Apr 28, 2015

@timesking It's brand new and will require some testing. I think it has a lot of potential.

@bep

This comment has been minimized.

Show comment
Hide comment
@bep

bep May 26, 2015

Member

Re. plugin framework, I think @natefinch got it conceptually right, see discussion at http://discuss.gohugo.io/t/using-for-plugins-in-hugo

Member

bep commented May 26, 2015

Re. plugin framework, I think @natefinch got it conceptually right, see discussion at http://discuss.gohugo.io/t/using-for-plugins-in-hugo

@lalloni

This comment has been minimized.

Show comment
Hide comment
@lalloni

lalloni May 27, 2015

@bep ... isn't that essentially what I proposed above?

Also, recently I've been exposed to Terraform's "plugin architecture" (I know that concept is not welcomed around here, sorry) which introduces a specific rpc protocol between parent & plugin processes and could be applicable too. See here.

lalloni commented May 27, 2015

@bep ... isn't that essentially what I proposed above?

Also, recently I've been exposed to Terraform's "plugin architecture" (I know that concept is not welcomed around here, sorry) which introduces a specific rpc protocol between parent & plugin processes and could be applicable too. See here.

@bep

This comment has been minimized.

Show comment
Hide comment
@bep

bep May 27, 2015

Member

@plalloni you are right, this is exactly what you outlined above. I have seen it mentioned before on the go-nuts group, so the idea isn't sparkling new -- but @natefinch is the first to put it into a well-tested (haven't looked at the coverage, but it looks fine) library.

There is still the question of security, though. If this should be of any value to the common Hugo user, he should be able to wire up his own plugins. He should be allowed to hang himself, but not be hanged by others ... so to speak.

Member

bep commented May 27, 2015

@plalloni you are right, this is exactly what you outlined above. I have seen it mentioned before on the go-nuts group, so the idea isn't sparkling new -- but @natefinch is the first to put it into a well-tested (haven't looked at the coverage, but it looks fine) library.

There is still the question of security, though. If this should be of any value to the common Hugo user, he should be able to wire up his own plugins. He should be allowed to hang himself, but not be hanged by others ... so to speak.

@ghostsquad

This comment has been minimized.

Show comment
Hide comment
@ghostsquad

ghostsquad Jul 17, 2015

👍 I would love to see Hugo Plugins.

👍 I would love to see Hugo Plugins.

@flyisland

This comment has been minimized.

Show comment
Hide comment
@flyisland

flyisland Jul 29, 2015

I do believe that text based diagram generators like plantuml, ditaa or dot are so powerful for technical content creator, and is a "must-have" feature for any static website generators.

For security concern, I think @halostatue 's idea is great:

This is something that the user must configure

Hugo will not and should not be responsible for any third-party plugin, the end user should download, install, enable and configure the third-party plugin by themselves. But Hugo would let the user know which plugins are about to run, and show the output message of these plugins to the user.

I do believe that text based diagram generators like plantuml, ditaa or dot are so powerful for technical content creator, and is a "must-have" feature for any static website generators.

For security concern, I think @halostatue 's idea is great:

This is something that the user must configure

Hugo will not and should not be responsible for any third-party plugin, the end user should download, install, enable and configure the third-party plugin by themselves. But Hugo would let the user know which plugins are about to run, and show the output message of these plugins to the user.

@uliska

This comment has been minimized.

Show comment
Hide comment
@uliska

uliska Aug 6, 2015

I admit I won't add anything substantial to the discussion but want to put some stress on the opinion that not having the option to call external commands in any way would really be a big minus, if not a show-stopper, for Hugo.

My use-case is similar to others described here: I am looking for a site generator that allows me to apply some custom syntax highlighting that is implemented in a Python script. And I want to include images generated from input code with an external program (namely LilyPond.

I also think that the suggestion above to make external commands available through a whitelist defined on project level would be a good compromise between security and flexibility.

uliska commented Aug 6, 2015

I admit I won't add anything substantial to the discussion but want to put some stress on the opinion that not having the option to call external commands in any way would really be a big minus, if not a show-stopper, for Hugo.

My use-case is similar to others described here: I am looking for a site generator that allows me to apply some custom syntax highlighting that is implemented in a Python script. And I want to include images generated from input code with an external program (namely LilyPond.

I also think that the suggestion above to make external commands available through a whitelist defined on project level would be a good compromise between security and flexibility.

@anthonyfok

This comment has been minimized.

Show comment
Hide comment
@anthonyfok

anthonyfok Aug 20, 2015

Member

My use-case is similar to others described here: I am looking for a site generator that allows me to apply some custom syntax highlighting that is implemented in a Python script. And I want to include images generated from input code with an external program (namely LilyPond).

So glad to see you here, @uliska! (I am a lurker on LilyPond mailing lists.)

You are right: the ability to use Hugo with LilyPond (and perhaps Gregorio too) would be heaven! :-)

Member

anthonyfok commented Aug 20, 2015

My use-case is similar to others described here: I am looking for a site generator that allows me to apply some custom syntax highlighting that is implemented in a Python script. And I want to include images generated from input code with an external program (namely LilyPond).

So glad to see you here, @uliska! (I am a lurker on LilyPond mailing lists.)

You are right: the ability to use Hugo with LilyPond (and perhaps Gregorio too) would be heaven! :-)

@uliska

This comment has been minimized.

Show comment
Hide comment
@uliska

uliska Aug 20, 2015

@anthonyfok Actually what I'm investigating is a solution for the documentation of (the "new") openLilyLib.

I found the plugin architecture of GitBook very nice where you can apply arbitrary JavaScript code on blocks of input - and this includes calling shell commands like the custom Python script as you can see here. Here you can see it in action (although in a very much sketchy document).
However, GitBook is so much targeted at a "book" style output that I can't use it for the purpose of a general website or even a generated documentation.

uliska commented Aug 20, 2015

@anthonyfok Actually what I'm investigating is a solution for the documentation of (the "new") openLilyLib.

I found the plugin architecture of GitBook very nice where you can apply arbitrary JavaScript code on blocks of input - and this includes calling shell commands like the custom Python script as you can see here. Here you can see it in action (although in a very much sketchy document).
However, GitBook is so much targeted at a "book" style output that I can't use it for the purpose of a general website or even a generated documentation.

@dnephin dnephin referenced this issue in docker/compose Nov 2, 2015

Closed

Generate command help from real output #2043

tubo28 added a commit to tubo28/hugo that referenced this issue Nov 4, 2015

Add command exec shortcode
Currently what it can is receive 2 arguments, the command and options. It inserts stdout to the called point. Flowing inner text into stdin is not now.
Whitelist is configurable and only cat (for unix) and type (for windows) are available in default.

See #796
@abourget

This comment has been minimized.

Show comment
Hide comment
@abourget

abourget Feb 24, 2016

Contributor

Folks, I'd like something like this too, for plantuml graphs..

Where as we on this ? Anything merged ?

Contributor

abourget commented Feb 24, 2016

Folks, I'd like something like this too, for plantuml graphs..

Where as we on this ? Anything merged ?

@bosr

This comment has been minimized.

Show comment
Hide comment

bosr commented Feb 24, 2016

👍

@abourget

This comment has been minimized.

Show comment
Hide comment
@abourget

abourget Feb 24, 2016

Contributor

I'd propose adding a whitelist of fully-qualified command paths that we are limited to passing on the command line or in the site config (with no possibility of override from themes).

ExecWhitelist:

  • /usr/bin/grep

and the exec call could simply use {{% exec grep %}} or something.. that would be resolved through the user's bin path resolution.

An "exec" shortcode could filter its content through exec.Command() - checking permissions first - using stdin/out.

I want that.. if I write it, can we merge it ? I want plantuml in there..

Please ! :)

Contributor

abourget commented Feb 24, 2016

I'd propose adding a whitelist of fully-qualified command paths that we are limited to passing on the command line or in the site config (with no possibility of override from themes).

ExecWhitelist:

  • /usr/bin/grep

and the exec call could simply use {{% exec grep %}} or something.. that would be resolved through the user's bin path resolution.

An "exec" shortcode could filter its content through exec.Command() - checking permissions first - using stdin/out.

I want that.. if I write it, can we merge it ? I want plantuml in there..

Please ! :)

@aliafshar

This comment has been minimized.

Show comment
Hide comment
@aliafshar

aliafshar Feb 24, 2016

Sorry, I have to basically strongly disagree with not allowing full shell access. A static site generator is being run by someone that has access to the shell, I don't understand the security rationale here. If anything I would love to be able to have my content as a result of shell pipes. javadoc | htmltidy | injectstyles etc etc. Whitelist is just too limited. This should be a core feature imho.

Note: I don't use Hugo because of this limitation.

Sorry, I have to basically strongly disagree with not allowing full shell access. A static site generator is being run by someone that has access to the shell, I don't understand the security rationale here. If anything I would love to be able to have my content as a result of shell pipes. javadoc | htmltidy | injectstyles etc etc. Whitelist is just too limited. This should be a core feature imho.

Note: I don't use Hugo because of this limitation.

@moorereason

This comment has been minimized.

Show comment
Hide comment
@moorereason

moorereason Feb 24, 2016

Member

@abourget,
I agree with @anthonyfok that we should add some kind of --unsafe command line flag. I'd hate to download somebody's repo to help them on an issue in the forums and there be malicious execs inside.

I go back and forth on the whitelist issue. SysAdmins should be able to modify environments to mitigate this in a service-oriented scenario. Can we use filepath.Glob to allow people to whitelist whole directories? RegExp would probably over-complicate this situation.

Member

moorereason commented Feb 24, 2016

@abourget,
I agree with @anthonyfok that we should add some kind of --unsafe command line flag. I'd hate to download somebody's repo to help them on an issue in the forums and there be malicious execs inside.

I go back and forth on the whitelist issue. SysAdmins should be able to modify environments to mitigate this in a service-oriented scenario. Can we use filepath.Glob to allow people to whitelist whole directories? RegExp would probably over-complicate this situation.

@bep

This comment has been minimized.

Show comment
Hide comment
@bep

bep Feb 24, 2016

Member

I have been hesitant about this from the start. @moorereason presents some good arguments. For me this is also a little bit related to the Hugo should delete /public before builds. This seems like a simple task to do right, but it's also a simple task to get horribly wrong in a multi-platform open source program. And even if it says "use as is and no warranties" in the LICENSE -- I'm reluctant to be the one who deletes valuable data on other peoples computers.

Member

bep commented Feb 24, 2016

I have been hesitant about this from the start. @moorereason presents some good arguments. For me this is also a little bit related to the Hugo should delete /public before builds. This seems like a simple task to do right, but it's also a simple task to get horribly wrong in a multi-platform open source program. And even if it says "use as is and no warranties" in the LICENSE -- I'm reluctant to be the one who deletes valuable data on other peoples computers.

@abourget

This comment has been minimized.

Show comment
Hide comment
@abourget

abourget Feb 24, 2016

Contributor

I'd be fine with an --unsafe flag.. and it is true that you execute the
program on the shell.

However, having a whitelist wouldmake things slightly more robust, and it
wouldn't prevent those wanting full shell access to write a simple
"do_awesome_pipes_magic.sh" and whitelist that.

Le mer. 24 févr. 2016 14:16, Bjørn Erik Pedersen notifications@github.com
a écrit :

I have been hesitant about this from the start. @moorereason
https://github.com/moorereason presents some good arguments. For me
this is also a little bit related to the Hugo should delete /public
before builds. This seems like a simple task to do right, but it's also a
simple task to get horribly wrong in a multi-platform open source program.
And even if it says "use as is and no warranties" in the LICENSE -- I'm
reluctant to be the one who deletes valuable data on other peoples
computers.


Reply to this email directly or view it on GitHub
#796 (comment).

Contributor

abourget commented Feb 24, 2016

I'd be fine with an --unsafe flag.. and it is true that you execute the
program on the shell.

However, having a whitelist wouldmake things slightly more robust, and it
wouldn't prevent those wanting full shell access to write a simple
"do_awesome_pipes_magic.sh" and whitelist that.

Le mer. 24 févr. 2016 14:16, Bjørn Erik Pedersen notifications@github.com
a écrit :

I have been hesitant about this from the start. @moorereason
https://github.com/moorereason presents some good arguments. For me
this is also a little bit related to the Hugo should delete /public
before builds. This seems like a simple task to do right, but it's also a
simple task to get horribly wrong in a multi-platform open source program.
And even if it says "use as is and no warranties" in the LICENSE -- I'm
reluctant to be the one who deletes valuable data on other peoples
computers.


Reply to this email directly or view it on GitHub
#796 (comment).

@christophermancini

This comment has been minimized.

Show comment
Hide comment
@christophermancini

christophermancini Mar 9, 2016

Contributor

👍 I would love plugin / cli execution functionality.

Contributor

christophermancini commented Mar 9, 2016

👍 I would love plugin / cli execution functionality.

@oasic

This comment has been minimized.

Show comment
Hide comment
@oasic

oasic Mar 18, 2016

I would have to agree with the others who have said this is a must have feature. I'm choosing MiddleMan right now for a project because this very important feature is lacking in Hugo.

oasic commented Mar 18, 2016

I would have to agree with the others who have said this is a must have feature. I'm choosing MiddleMan right now for a project because this very important feature is lacking in Hugo.

@danielbarbarito

This comment has been minimized.

Show comment
Hide comment
@danielbarbarito

danielbarbarito Jul 30, 2016

+1 I would love a feature like this

+1 I would love a feature like this

@abourget

This comment has been minimized.

Show comment
Hide comment
@abourget

abourget Oct 12, 2016

Contributor

Do we have agreement on --unsafe-exec for example ? If so, I'll implement it and propose it here. I need that, in the past, now and in the future.

Contributor

abourget commented Oct 12, 2016

Do we have agreement on --unsafe-exec for example ? If so, I'll implement it and propose it here. I need that, in the past, now and in the future.

@bep

This comment has been minimized.

Show comment
Hide comment
@bep

bep Oct 12, 2016

Member

Do we have agreement on --unsafe-exec for example

No.

Member

bep commented Oct 12, 2016

Do we have agreement on --unsafe-exec for example

No.

@moorereason

This comment has been minimized.

Show comment
Hide comment
@moorereason

moorereason Oct 12, 2016

Member

I know plugins came up earlier in this discussion, but that's a whole different animal. I'm ignoring plugins for now.

My proposal for an exec feature:

  • Add --unsafe-exec command-line option that defaults to false. No matching site config option.
  • Add execWhitelist (a slice of file.Match patterns) to the site config.
  • Add exec template function and shortcode.
  • Add allowThemeExec bool to site config to allow exec usage in themes. Default is false.
  • Using exec without --unsafe-exec and a matching execWhitelist entry generates a fatal build error.
  • Using exec in a theme without --unsafe-exec, a matching execWhitelist entry, and allowThemeExec enabled generates a fatal build error.

Example whitelist config:

execWhitelist = [
    "/opt/mytools/bin/*",  # absolute path
    "curl",                # searches $PATH
    "*",                   # you don't care about security

   "bin/*",                # error here? relative path (contains path separator);
                           # must be bare command or absolute path?
]

My main concerns with exec:

  • Cloning someone's site to troubleshoot. Answer: don't use --unsafe-exec.
  • Theme authors can hurt me. Answer: don't use --unsafe-exec. If you want to use exec in your layouts, don't enable allowThemeExec.
  • Hosting providers could be affected. Providers, don't allow your customers to pass --unsafe-exec in the Hugo build command and/or don't allow them to upload binaries in their site. Secure your environment, otherwise.
Member

moorereason commented Oct 12, 2016

I know plugins came up earlier in this discussion, but that's a whole different animal. I'm ignoring plugins for now.

My proposal for an exec feature:

  • Add --unsafe-exec command-line option that defaults to false. No matching site config option.
  • Add execWhitelist (a slice of file.Match patterns) to the site config.
  • Add exec template function and shortcode.
  • Add allowThemeExec bool to site config to allow exec usage in themes. Default is false.
  • Using exec without --unsafe-exec and a matching execWhitelist entry generates a fatal build error.
  • Using exec in a theme without --unsafe-exec, a matching execWhitelist entry, and allowThemeExec enabled generates a fatal build error.

Example whitelist config:

execWhitelist = [
    "/opt/mytools/bin/*",  # absolute path
    "curl",                # searches $PATH
    "*",                   # you don't care about security

   "bin/*",                # error here? relative path (contains path separator);
                           # must be bare command or absolute path?
]

My main concerns with exec:

  • Cloning someone's site to troubleshoot. Answer: don't use --unsafe-exec.
  • Theme authors can hurt me. Answer: don't use --unsafe-exec. If you want to use exec in your layouts, don't enable allowThemeExec.
  • Hosting providers could be affected. Providers, don't allow your customers to pass --unsafe-exec in the Hugo build command and/or don't allow them to upload binaries in their site. Secure your environment, otherwise.
@abourget

This comment has been minimized.

Show comment
Hide comment
@abourget

abourget Oct 12, 2016

Contributor

Follow up on discussion in https://gitter.im/spf13/hugo today:

  1. use cases (GraphViz/LilyPond/pygments/PlantUML)
  2. security (explicit flag --unsafe-slow-exec, on a tool the admin runs explicitly on the CLI)
  3. forum management (with slow in the flag)
  4. simple implementation, implementing a simple yet flexible protocol (proposed by this Issue implemented by #847 + some refinements abouts flags.)

Flags in the form of:

  • --unsafe-slow-exec /usr/bin/exec1,/usr/bin/exec2 or
  • --unsafe-slow-exec /usr/bin/exec1 --unsafe-slow-exec /bin/exec2

Quite a few people are interested into Hooks or plugins in a more Go-like fashion.. Perhaps we can have another Issue for that, as this one is related to an exec feature.

There has been concerns about shareability of pieces by content authors, but this Issue doesn't address this. It is more for featurefullness.

Contributor

abourget commented Oct 12, 2016

Follow up on discussion in https://gitter.im/spf13/hugo today:

  1. use cases (GraphViz/LilyPond/pygments/PlantUML)
  2. security (explicit flag --unsafe-slow-exec, on a tool the admin runs explicitly on the CLI)
  3. forum management (with slow in the flag)
  4. simple implementation, implementing a simple yet flexible protocol (proposed by this Issue implemented by #847 + some refinements abouts flags.)

Flags in the form of:

  • --unsafe-slow-exec /usr/bin/exec1,/usr/bin/exec2 or
  • --unsafe-slow-exec /usr/bin/exec1 --unsafe-slow-exec /bin/exec2

Quite a few people are interested into Hooks or plugins in a more Go-like fashion.. Perhaps we can have another Issue for that, as this one is related to an exec feature.

There has been concerns about shareability of pieces by content authors, but this Issue doesn't address this. It is more for featurefullness.

@abourget

This comment has been minimized.

Show comment
Hide comment
@abourget

abourget Oct 12, 2016

Contributor

Perhaps we can think of different ways (config.ini ?) to pass those whitelisted binaries ?

Contributor

abourget commented Oct 12, 2016

Perhaps we can think of different ways (config.ini ?) to pass those whitelisted binaries ?

@lamvak

This comment has been minimized.

Show comment
Hide comment
@lamvak

lamvak Oct 24, 2016

I applaud the level of concern given to security. I'm sorry for the lengt of the below - I really tried to boil it down. I hope it's still worth your time.

Is there any documentation regarding security model for Hugo? Say, overview of modules, actors, deployment model like target platforms (software meant for local development desktop rather than production server), etc., as well as description of security responsibilities or assumptions for those; i.e. security defaults, or explaining concept of keeping 'serve' module secure by listening only on local interface, etc.)
With a general guidelines, definition of such a document could be driven by QnA from devs/contributors/users.

The themes are understood as separate software components, which is evidenced in liberties given to themes developers and users, as well as separate themes licensing.
I would propose that in essence the Hugo themes user (person installing them) is ultimately responsible for their secure use. User must either understand the software being installed - or trust the theme development process (i.e. community with peer review) - no innovation here.

Theme author together with theme users are final arbiters of what code/use cases are sensible for extended complex additional theme programming - so far as no one feels entitled to have their code pulled into Hugo. This means I can ship a theme today with a shell script providing some preprocessing to the content files and users may choose to use the theme. While it makes sense that you may decline support for such theme and may wish not to use it yourself, would you go so far as to forbid the practice?
If you wouldn't, issue boils down not to having arbitrary code in themes, but to have Hugo responsible for triggering it. If you accept as I propose above that it's ultimately users' responsibility to understand what software is in use, the question shifts to one of balance between securing the use of Hugo on behalf of the users vs flexibility and ease of use for the custom code.

Additionally, a point that relates not only to plugins but the mentioned Hugo features like removing data from public directory: Would it be more productive to solve security issues in the actual security domain? I.e. do not run as priviledged user, do not run on production servers; if feasible - run as dedicated user with limited resources access, or even in chroot-like environment. Of course, this is not necessarily something to be built-in inside of Hugo.

lamvak commented Oct 24, 2016

I applaud the level of concern given to security. I'm sorry for the lengt of the below - I really tried to boil it down. I hope it's still worth your time.

Is there any documentation regarding security model for Hugo? Say, overview of modules, actors, deployment model like target platforms (software meant for local development desktop rather than production server), etc., as well as description of security responsibilities or assumptions for those; i.e. security defaults, or explaining concept of keeping 'serve' module secure by listening only on local interface, etc.)
With a general guidelines, definition of such a document could be driven by QnA from devs/contributors/users.

The themes are understood as separate software components, which is evidenced in liberties given to themes developers and users, as well as separate themes licensing.
I would propose that in essence the Hugo themes user (person installing them) is ultimately responsible for their secure use. User must either understand the software being installed - or trust the theme development process (i.e. community with peer review) - no innovation here.

Theme author together with theme users are final arbiters of what code/use cases are sensible for extended complex additional theme programming - so far as no one feels entitled to have their code pulled into Hugo. This means I can ship a theme today with a shell script providing some preprocessing to the content files and users may choose to use the theme. While it makes sense that you may decline support for such theme and may wish not to use it yourself, would you go so far as to forbid the practice?
If you wouldn't, issue boils down not to having arbitrary code in themes, but to have Hugo responsible for triggering it. If you accept as I propose above that it's ultimately users' responsibility to understand what software is in use, the question shifts to one of balance between securing the use of Hugo on behalf of the users vs flexibility and ease of use for the custom code.

Additionally, a point that relates not only to plugins but the mentioned Hugo features like removing data from public directory: Would it be more productive to solve security issues in the actual security domain? I.e. do not run as priviledged user, do not run on production servers; if feasible - run as dedicated user with limited resources access, or even in chroot-like environment. Of course, this is not necessarily something to be built-in inside of Hugo.

@bep

This comment has been minimized.

Show comment
Hide comment
@bep

bep Oct 24, 2016

Member

Is there any documentation regarding security model for Hugo?

No. But it is important to think of a "security model" as to mostly concerning the well being of the "Hugo user's computer and data" and not mainly his or hers Hugo site (which is important enough).

It is a balance.

But for a given feature I ask myself:

  • Is it possible to implement this in a safe way without too much complexity?
  • Do we have any contributors that can implement it this way? I.e. solid, robust and fully tested.
  • Are there still chances that, even if the initial implementation was solid, an innocent contribution by someone else changes the meaning of the flag enableExecPlugin (maybe in another library, such as Viper)?
  • Even if all the above checks out: Is the feature worth the amount of work needed compared to its value and other features in the pipeline?

And the answer to the last one is currently no. There are currently a lot of other nice stuff to implement before a very ineffective plugin model (if you don't understand how os/exec works, read up about it and its use of hardware threads etc.), and in the big picture I don't see many people really wanting it.

And remember that features have to be maintained and supported, and I currently don't see anyone in this thread that is able to implement this in a secure way and at the same time is willing to maintain and support it.

Member

bep commented Oct 24, 2016

Is there any documentation regarding security model for Hugo?

No. But it is important to think of a "security model" as to mostly concerning the well being of the "Hugo user's computer and data" and not mainly his or hers Hugo site (which is important enough).

It is a balance.

But for a given feature I ask myself:

  • Is it possible to implement this in a safe way without too much complexity?
  • Do we have any contributors that can implement it this way? I.e. solid, robust and fully tested.
  • Are there still chances that, even if the initial implementation was solid, an innocent contribution by someone else changes the meaning of the flag enableExecPlugin (maybe in another library, such as Viper)?
  • Even if all the above checks out: Is the feature worth the amount of work needed compared to its value and other features in the pipeline?

And the answer to the last one is currently no. There are currently a lot of other nice stuff to implement before a very ineffective plugin model (if you don't understand how os/exec works, read up about it and its use of hardware threads etc.), and in the big picture I don't see many people really wanting it.

And remember that features have to be maintained and supported, and I currently don't see anyone in this thread that is able to implement this in a secure way and at the same time is willing to maintain and support it.

@bep

This comment has been minimized.

Show comment
Hide comment
@bep

bep Oct 24, 2016

Member

But please take further discussion at http://discuss.gohugo.io/

Member

bep commented Oct 24, 2016

But please take further discussion at http://discuss.gohugo.io/

@gohugoio gohugoio locked and limited conversation to collaborators Oct 24, 2016

@bep bep added the Stale label Jul 1, 2017

@bep

This comment has been minimized.

Show comment
Hide comment
@bep

bep Jul 1, 2017

Member

This issue has been automatically marked as stale because it has not been commented on for at least six months.

The resources of the Hugo team are limited, and so we are asking for your help.

If this is a bug and you can still reproduce this error on the master branch, please reply with all of the information you have about it in order to keep the issue open.

If this is a feature request, and you feel that it is still relevant and valuable, please tell us why.

This issue will automatically be closed in the near future if no further activity occurs. Thank you for all your contributions.

Member

bep commented Jul 1, 2017

This issue has been automatically marked as stale because it has not been commented on for at least six months.

The resources of the Hugo team are limited, and so we are asking for your help.

If this is a bug and you can still reproduce this error on the master branch, please reply with all of the information you have about it in order to keep the issue open.

If this is a feature request, and you feel that it is still relevant and valuable, please tell us why.

This issue will automatically be closed in the near future if no further activity occurs. Thank you for all your contributions.

@moorereason moorereason added Keep and removed Stale labels Nov 7, 2017

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.