Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add exec shortcode #796

Open
bep opened this issue Jan 14, 2015 · 53 comments

Comments

Projects
None yet
@bep
Copy link
Member

commented Jan 14, 2015

Idea from this post:
http://discuss.gohugo.io/t/inline-ditaa-dot-or-plantuml-sources-in-hugo-posts/589/3

Similar to Highlight. but for any binary that writes to stdout.

Two arguments:

  1. Binary
  2. Args
  • the inner content to stdin

I don't see any obvious issues why we could/should not add this? Security issues? It's up to the developer to use or abuse this?

@bep bep added the Enhancement label Jan 14, 2015

@bep

This comment has been minimized.

Copy link
Member Author

commented Jan 15, 2015

Come to think of it, it will give theme-authors the opening to do harmful stuff ... like "rm -r /".

But it's an interesting thought - if we could restrict it to a set of commands.

Just tested cat combined with highlight and some go files. Very useful.

{{< highlight go >}}
{{< exec "cat" "/home/bep/dev/go/src/github.com/spf13/hugo/hugolib/shortcodeparser.go" >}}
{{< / highlight >}}

Note, the above will not make any sense without the fix for #797 - but it's kind of cool.

@halostatue

This comment has been minimized.

Copy link
Contributor

commented Jan 15, 2015

I don’t think it gives theme authors anything—shortcodes aren’t rendered by templates, only content. Right?

@anthonyfok

This comment has been minimized.

Copy link
Contributor

commented Jan 15, 2015

Interesting, and very useful indeed! :-)

Though, for security reasons, I think it should be a configurable option that is turned off by default.
It may also be helpful to print out the exec command for each website rebuild so that the end user is fully aware of what is going on. And yes, restriction to a set of commands (and other needed sanitization) would be a good thing.

@bep

This comment has been minimized.

Copy link
Member Author

commented Jan 15, 2015

@halostatue no, but shortcodes can be provided by themes. So a shortcode from a theme named coolinnocentshortcode is tempting to use.

But I think this is too useful turn down without some thinking.

EDIT IN: I may be wrong -- if we restrict this to shortcodes only, this will only be available to content files ...

@bep

This comment has been minimized.

Copy link
Member Author

commented Jan 15, 2015

But it would be cool if we could make some of the obvious available as both template function and shortcode (if they exist on the computer):

  • cat
  • tail
  • curl
  • ??
@spf13

This comment has been minimized.

Copy link
Contributor

commented Jan 16, 2015

Sorry I'm a bit late to this. When I saw this I immediately thought that it was a security issue as you pointed out in your follow up.

I agree that the best approach would be to define a few commonly used commands.

cat & grep alone would be super powerful. Curl may be but could also pose a security risk.

Another concern is that these won't work for Windows.

@bep

This comment has been minimized.

Copy link
Member Author

commented Jan 16, 2015

Yea, thinking a little about this. I foresee a support nightmare. "I get 'broken pipe' with Photoshop".

I might maintain a private patch for myself, but for Hugo it's maybe better to find a Go implementation similar to cat etc.

My current use case is what Martin Fowler calls "live code" in documentation - but much simpler than his setup, i. e. text extracts from source code on disk (I know about Gists).

@rodlogic

This comment has been minimized.

Copy link

commented Jan 17, 2015

My main motivation for this is so I can plug in tools such as plantuml, ditaa or graphviz to generate diagrams while building the site (Jekyll supports this quite well). AFAIK, this is not possible in Hugo at this point and that a big minus to me.

Although having curl, cat etc as standard shortcodes, it would not address what I am looking for. Either there is a generic exec short code or a specific one for these tools above (here a plugin approach would be best specially since any support questions can be deferred to the developer of the plugin).

@bep

This comment has been minimized.

Copy link
Member Author

commented Jan 17, 2015

Go's static linking is mostly a blessing, but it also kicks the feet under the plugin word.

Go has a brilliant and simple exec-package that is perfect for this purpose, but I don't see Hugo pulling in plantuml, ditaa ... as specific shortcodes/functions like Pygments.

Maybe if a general Exec could be secured (whitelist?)

@lalloni

This comment has been minimized.

Copy link

commented Jan 25, 2015

I've just found this PR while searching for some way of using plantuml from hugo and I must say that this looks almost exactly like what we need. Thanks!

WRT a possible plugin infrastructure in hugo/go, how about a plugin architecture in which plugins are standalone programs which expect arguments from stdin in some standard form (json, protobuf, etc.) and write answers to stdout using the same format (with possible secondary effects in the output fs directory), this plugins should be registered at the project level (not from themes) and invoked from content or templates.

@bep

This comment has been minimized.

Copy link
Member Author

commented Jan 25, 2015

@plalloni well, the "plugin architecure" you want is basically a whitelist for the "exec template function", but who approves the whitelist?

There are great powers in Brad F. something's exec package, but as stated, there may be concerns open it up, even to a static site gen.

@lalloni

This comment has been minimized.

Copy link

commented Jan 25, 2015

@bjornerik Why is a whitelist needed if plugins are not to be registered by theme creators but only by site authors (i.e. a theme could require a certain plugin installed but can not install it by itself, so the site/project owner would be required to install/register the plugin).

@spf13

This comment has been minimized.

Copy link
Contributor

commented Jan 25, 2015

@plalloni There aren't "plugins". These would be methods for the template system (which would call exec) and there's not currently any way of restricting them from some templates and not others. Consequently we would need to keep it clean everywhere.

@bep bep added the discussion label Jan 27, 2015

@lalloni

This comment has been minimized.

Copy link

commented Jan 27, 2015

@bjornerik Any chance I can get my hands on your patch?

@bep

This comment has been minimized.

Copy link
Member Author

commented Jan 27, 2015

@plalloni it's not in a shareable state.

@aliafshar

This comment has been minimized.

Copy link

commented Jan 27, 2015

Limited range of commands won't work for me. This would be a great hook for running arbitrary external commands to perform transforms on part of the content. I will write a patch for my own use.

@halostatue

This comment has been minimized.

Copy link
Contributor

commented Jan 27, 2015

I think that this can still be written in a way as to be safe, using front matter or config.*. If I do:

exec.whitelist = [
  "cat",
  "tac",
  "curl"
]

This is something that the user must configure, and any command where the command does not exactly match a string in the whitelist will fail to run.

@aliafshar

This comment has been minimized.

Copy link

commented Jan 27, 2015

Here is my patch #847

Nothing fancy, adds an exec function that will be usable in shortcodes I hope. Usable for my purpose, but needs work for any thoughts of inclusion into the main tree.

@lalloni

This comment has been minimized.

Copy link

commented Jan 28, 2015

@aliafshar Thank you! I'll try it ASAP.

@dcorb

This comment has been minimized.

Copy link

commented Feb 5, 2015

I would love to see this feature in Hugo.
This opens the doors to all kind of external utilities. From basic 'cat' to generating charts with d3js. Something like this:
http://www.scottlogic.com/blog/2014/09/15/jekyll-d3js.html

Security is here the biggest concern, but this would be so useful.

As a follow up: Would be possible to cache "exec" outputs, to prevent slowing down the system?

@lalloni

This comment has been minimized.

Copy link

commented Feb 6, 2015

WRT security: what if allowed executables must be in some specific site level path like /bin (where can't be installed by theme creators)?

Site owner can symlink from bin to anywhere in the system, and/or install specific programs or scripts for calling tools like plantuml o d3.

@anthonyfok

This comment has been minimized.

Copy link
Contributor

commented Feb 6, 2015

Or, how about this:

Normally, when hugo is run with no special flags, the exec statements are printed but not executed;
The exec statements are only run when the option --unsafe is added?

For added safety measure, even with --unsafe, perhaps Hugo would pause and list all exec statements, asking the user whether they are sure if they want to proceed (y/N)? And perhaps an added flag --yes-i-really-know-what-i-am-doing is needed to skip that question?

@aliafshar

This comment has been minimized.

Copy link

commented Feb 9, 2015

And, I changed my mind here. I am running an independent preprocessor on the content. I think the main problem here is that hugo will at some point need a way of being extended with plugins that execute code, not just template shortcodes - makes long-term sense for a system like this.

Then security can be addressed by explicitly allowing plugins. I appreciate Golang is going to make this very hard.

@lalloni

This comment has been minimized.

Copy link

commented Feb 9, 2015

@aliafshar I agree. I've been thinking about "plugins" as executable files/links somewhere in the site directory structure. Hugo could run them as subprocesses and comunicate with them using stdin/out. Could be one-off subprocesses or "server style" processes (for keeping latency as low as possible) attending many requests.

@timesking

This comment has been minimized.

Copy link

commented Apr 28, 2015

How about this plugin framework, https://github.com/dullgiulio/pingo

@spf13

This comment has been minimized.

Copy link
Contributor

commented Apr 28, 2015

@timesking It's brand new and will require some testing. I think it has a lot of potential.

@bep

This comment has been minimized.

Copy link
Member Author

commented May 26, 2015

Re. plugin framework, I think @natefinch got it conceptually right, see discussion at http://discuss.gohugo.io/t/using-for-plugins-in-hugo

@lalloni

This comment has been minimized.

Copy link

commented May 27, 2015

@bep ... isn't that essentially what I proposed above?

Also, recently I've been exposed to Terraform's "plugin architecture" (I know that concept is not welcomed around here, sorry) which introduces a specific rpc protocol between parent & plugin processes and could be applicable too. See here.

@anthonyfok

This comment has been minimized.

Copy link
Contributor

commented Aug 20, 2015

My use-case is similar to others described here: I am looking for a site generator that allows me to apply some custom syntax highlighting that is implemented in a Python script. And I want to include images generated from input code with an external program (namely LilyPond).

So glad to see you here, @uliska! (I am a lurker on LilyPond mailing lists.)

You are right: the ability to use Hugo with LilyPond (and perhaps Gregorio too) would be heaven! :-)

@uliska

This comment has been minimized.

Copy link

commented Aug 20, 2015

@anthonyfok Actually what I'm investigating is a solution for the documentation of (the "new") openLilyLib.

I found the plugin architecture of GitBook very nice where you can apply arbitrary JavaScript code on blocks of input - and this includes calling shell commands like the custom Python script as you can see here. Here you can see it in action (although in a very much sketchy document).
However, GitBook is so much targeted at a "book" style output that I can't use it for the purpose of a general website or even a generated documentation.

tubo28 added a commit to tubo28/hugo that referenced this issue Nov 4, 2015

Add command exec shortcode
Currently what it can is receive 2 arguments, the command and options. It inserts stdout to the called point. Flowing inner text into stdin is not now.
Whitelist is configurable and only cat (for unix) and type (for windows) are available in default.

See gohugoio#796
@abourget

This comment has been minimized.

Copy link
Contributor

commented Feb 24, 2016

Folks, I'd like something like this too, for plantuml graphs..

Where as we on this ? Anything merged ?

@bosr

This comment has been minimized.

Copy link

commented Feb 24, 2016

👍

@abourget

This comment has been minimized.

Copy link
Contributor

commented Feb 24, 2016

I'd propose adding a whitelist of fully-qualified command paths that we are limited to passing on the command line or in the site config (with no possibility of override from themes).

ExecWhitelist:

  • /usr/bin/grep

and the exec call could simply use {{% exec grep %}} or something.. that would be resolved through the user's bin path resolution.

An "exec" shortcode could filter its content through exec.Command() - checking permissions first - using stdin/out.

I want that.. if I write it, can we merge it ? I want plantuml in there..

Please ! :)

@aliafshar

This comment has been minimized.

Copy link

commented Feb 24, 2016

Sorry, I have to basically strongly disagree with not allowing full shell access. A static site generator is being run by someone that has access to the shell, I don't understand the security rationale here. If anything I would love to be able to have my content as a result of shell pipes. javadoc | htmltidy | injectstyles etc etc. Whitelist is just too limited. This should be a core feature imho.

Note: I don't use Hugo because of this limitation.

@moorereason

This comment has been minimized.

Copy link
Contributor

commented Feb 24, 2016

@abourget,
I agree with @anthonyfok that we should add some kind of --unsafe command line flag. I'd hate to download somebody's repo to help them on an issue in the forums and there be malicious execs inside.

I go back and forth on the whitelist issue. SysAdmins should be able to modify environments to mitigate this in a service-oriented scenario. Can we use filepath.Glob to allow people to whitelist whole directories? RegExp would probably over-complicate this situation.

@bep

This comment has been minimized.

Copy link
Member Author

commented Feb 24, 2016

I have been hesitant about this from the start. @moorereason presents some good arguments. For me this is also a little bit related to the Hugo should delete /public before builds. This seems like a simple task to do right, but it's also a simple task to get horribly wrong in a multi-platform open source program. And even if it says "use as is and no warranties" in the LICENSE -- I'm reluctant to be the one who deletes valuable data on other peoples computers.

@abourget

This comment has been minimized.

Copy link
Contributor

commented Feb 24, 2016

I'd be fine with an --unsafe flag.. and it is true that you execute the
program on the shell.

However, having a whitelist wouldmake things slightly more robust, and it
wouldn't prevent those wanting full shell access to write a simple
"do_awesome_pipes_magic.sh" and whitelist that.

Le mer. 24 févr. 2016 14:16, Bjørn Erik Pedersen notifications@github.com
a écrit :

I have been hesitant about this from the start. @moorereason
https://github.com/moorereason presents some good arguments. For me
this is also a little bit related to the Hugo should delete /public
before builds. This seems like a simple task to do right, but it's also a
simple task to get horribly wrong in a multi-platform open source program.
And even if it says "use as is and no warranties" in the LICENSE -- I'm
reluctant to be the one who deletes valuable data on other peoples
computers.


Reply to this email directly or view it on GitHub
#796 (comment).

@christophermancini

This comment has been minimized.

Copy link
Contributor

commented Mar 9, 2016

👍 I would love plugin / cli execution functionality.

@oasic

This comment has been minimized.

Copy link

commented Mar 18, 2016

I would have to agree with the others who have said this is a must have feature. I'm choosing MiddleMan right now for a project because this very important feature is lacking in Hugo.

@danielbarbarito

This comment has been minimized.

Copy link

commented Jul 30, 2016

+1 I would love a feature like this

@abourget

This comment has been minimized.

Copy link
Contributor

commented Oct 12, 2016

Do we have agreement on --unsafe-exec for example ? If so, I'll implement it and propose it here. I need that, in the past, now and in the future.

@bep

This comment has been minimized.

Copy link
Member Author

commented Oct 12, 2016

Do we have agreement on --unsafe-exec for example

No.

@moorereason

This comment has been minimized.

Copy link
Contributor

commented Oct 12, 2016

I know plugins came up earlier in this discussion, but that's a whole different animal. I'm ignoring plugins for now.

My proposal for an exec feature:

  • Add --unsafe-exec command-line option that defaults to false. No matching site config option.
  • Add execWhitelist (a slice of file.Match patterns) to the site config.
  • Add exec template function and shortcode.
  • Add allowThemeExec bool to site config to allow exec usage in themes. Default is false.
  • Using exec without --unsafe-exec and a matching execWhitelist entry generates a fatal build error.
  • Using exec in a theme without --unsafe-exec, a matching execWhitelist entry, and allowThemeExec enabled generates a fatal build error.

Example whitelist config:

execWhitelist = [
    "/opt/mytools/bin/*",  # absolute path
    "curl",                # searches $PATH
    "*",                   # you don't care about security

   "bin/*",                # error here? relative path (contains path separator);
                           # must be bare command or absolute path?
]

My main concerns with exec:

  • Cloning someone's site to troubleshoot. Answer: don't use --unsafe-exec.
  • Theme authors can hurt me. Answer: don't use --unsafe-exec. If you want to use exec in your layouts, don't enable allowThemeExec.
  • Hosting providers could be affected. Providers, don't allow your customers to pass --unsafe-exec in the Hugo build command and/or don't allow them to upload binaries in their site. Secure your environment, otherwise.
@abourget

This comment has been minimized.

Copy link
Contributor

commented Oct 12, 2016

Follow up on discussion in https://gitter.im/spf13/hugo today:

  1. use cases (GraphViz/LilyPond/pygments/PlantUML)
  2. security (explicit flag --unsafe-slow-exec, on a tool the admin runs explicitly on the CLI)
  3. forum management (with slow in the flag)
  4. simple implementation, implementing a simple yet flexible protocol (proposed by this Issue implemented by #847 + some refinements abouts flags.)

Flags in the form of:

  • --unsafe-slow-exec /usr/bin/exec1,/usr/bin/exec2 or
  • --unsafe-slow-exec /usr/bin/exec1 --unsafe-slow-exec /bin/exec2

Quite a few people are interested into Hooks or plugins in a more Go-like fashion.. Perhaps we can have another Issue for that, as this one is related to an exec feature.

There has been concerns about shareability of pieces by content authors, but this Issue doesn't address this. It is more for featurefullness.

@abourget

This comment has been minimized.

Copy link
Contributor

commented Oct 12, 2016

Perhaps we can think of different ways (config.ini ?) to pass those whitelisted binaries ?

@lamvak

This comment has been minimized.

Copy link

commented Oct 24, 2016

I applaud the level of concern given to security. I'm sorry for the lengt of the below - I really tried to boil it down. I hope it's still worth your time.

Is there any documentation regarding security model for Hugo? Say, overview of modules, actors, deployment model like target platforms (software meant for local development desktop rather than production server), etc., as well as description of security responsibilities or assumptions for those; i.e. security defaults, or explaining concept of keeping 'serve' module secure by listening only on local interface, etc.)
With a general guidelines, definition of such a document could be driven by QnA from devs/contributors/users.

The themes are understood as separate software components, which is evidenced in liberties given to themes developers and users, as well as separate themes licensing.
I would propose that in essence the Hugo themes user (person installing them) is ultimately responsible for their secure use. User must either understand the software being installed - or trust the theme development process (i.e. community with peer review) - no innovation here.

Theme author together with theme users are final arbiters of what code/use cases are sensible for extended complex additional theme programming - so far as no one feels entitled to have their code pulled into Hugo. This means I can ship a theme today with a shell script providing some preprocessing to the content files and users may choose to use the theme. While it makes sense that you may decline support for such theme and may wish not to use it yourself, would you go so far as to forbid the practice?
If you wouldn't, issue boils down not to having arbitrary code in themes, but to have Hugo responsible for triggering it. If you accept as I propose above that it's ultimately users' responsibility to understand what software is in use, the question shifts to one of balance between securing the use of Hugo on behalf of the users vs flexibility and ease of use for the custom code.

Additionally, a point that relates not only to plugins but the mentioned Hugo features like removing data from public directory: Would it be more productive to solve security issues in the actual security domain? I.e. do not run as priviledged user, do not run on production servers; if feasible - run as dedicated user with limited resources access, or even in chroot-like environment. Of course, this is not necessarily something to be built-in inside of Hugo.

@bep

This comment has been minimized.

Copy link
Member Author

commented Oct 24, 2016

Is there any documentation regarding security model for Hugo?

No. But it is important to think of a "security model" as to mostly concerning the well being of the "Hugo user's computer and data" and not mainly his or hers Hugo site (which is important enough).

It is a balance.

But for a given feature I ask myself:

  • Is it possible to implement this in a safe way without too much complexity?
  • Do we have any contributors that can implement it this way? I.e. solid, robust and fully tested.
  • Are there still chances that, even if the initial implementation was solid, an innocent contribution by someone else changes the meaning of the flag enableExecPlugin (maybe in another library, such as Viper)?
  • Even if all the above checks out: Is the feature worth the amount of work needed compared to its value and other features in the pipeline?

And the answer to the last one is currently no. There are currently a lot of other nice stuff to implement before a very ineffective plugin model (if you don't understand how os/exec works, read up about it and its use of hardware threads etc.), and in the big picture I don't see many people really wanting it.

And remember that features have to be maintained and supported, and I currently don't see anyone in this thread that is able to implement this in a secure way and at the same time is willing to maintain and support it.

@bep

This comment has been minimized.

Copy link
Member Author

commented Oct 24, 2016

But please take further discussion at http://discuss.gohugo.io/

@gohugoio gohugoio locked and limited conversation to collaborators Oct 24, 2016

@bep bep added the Stale label Jul 1, 2017

@bep

This comment has been minimized.

Copy link
Member Author

commented Jul 1, 2017

This issue has been automatically marked as stale because it has not been commented on for at least six months.

The resources of the Hugo team are limited, and so we are asking for your help.

If this is a bug and you can still reproduce this error on the master branch, please reply with all of the information you have about it in order to keep the issue open.

If this is a feature request, and you feel that it is still relevant and valuable, please tell us why.

This issue will automatically be closed in the near future if no further activity occurs. Thank you for all your contributions.

@moorereason moorereason added Keep and removed Stale labels Nov 7, 2017

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.