add org content support with org keyword headers as frontmatter

[curiouslychase]

This PR adds support for Org content, as requested in #936 and #1483.

It doesn't look like much, but it adds a new dependency https://github.com/chaseadamsio/goorgeous, a go org to html package.

Files with the following format with .org extensions are supported for 98% of the use cases and 100% of what would be converted from Hugo markdown currently.

It supports .org headers at the top of the file as well for frontmatter:

#+TITLE: My Hugo Blog Post
#+AUTHOR: Chase Adams
#+DATE: 2016-12-14T10:17:43Z
#+DRAFT: true

so there's no need to "manipulate" org files to work with Hugo by including toml, yaml or JSON for frontmatter.

It'd be great to have some folks test it out (I've been generating posts for the past two weeks with this support successfully).

Any guidance on what tests should be written to verify and what documentation should be updated would be appreciated!

@bep @spf13

cc @anthonyfok @eggcaker

[bep]

This looks good, I'm not familiar with org mode, but I understand that many will find this very useful.

As to tests; I will have a closer look at this in a couple of days -- when I get the current PR I'm working on into master. It will not touch your code, but it will make some changes in how tests are written, so it would be good for you to wait with those until then.

[curiouslychase]

Cool. Thanks @bep.

[bep]

One first comment would be to add a compatible license to goorgeous so we can acutally use that library.

[curiouslychase]

Added MIT license. curiouslychase/goorgeous@590ae02

[digitalcraftsman]

I haven't used org-mode myself but read good things about it. Having another markup language to choose from is great, especially with a Go-native implementation 👍

Any guidance on what tests should be written to verify and what documentation should be updated would be appreciated!

Regarding the documentation I would suggest to mention the org-mode support where the other markup formats are also listed/mentioned. Just search in the repo for asciidoc.

[curiouslychase]

@bep any update on the test PR you referenced previously?

[bep]

No. Or, it is getting closer to finish.

[bep]

It is this PR: #2932

I just finished it -- gonna let all the tests go to green.

[bep]

The main test I think we need would be to add org to the list here:

https://github.com/spf13/hugo/blob/master/hugolib/page_test.go#L595

You would have to do more work than that, I guess. It this doesn't make sense to put in here, or it gets too ugly, you would have to make a separate table driven test that does approximately the same.

But I guess it would be possible to make the frontmatter variants into funcs that checks what markdown to use etc.

[bep]

Thinking about it, maybe it would be better if you could just create a separate page_org_test.go or something. Not sure. Up to you.

But the pattern of the tests are shown:

1. Create a fs and write some source variations to it.
2. Do a s := buildSingleSite(t, deps.DepsCfg{Fs: fs}, BuildCfg{SkipRender: true})
3. Do assertions in the site's pages.

But the main point is: I trust that you test your own library, so focus on the few Hugo specifics:

• Summary handling etc.
• Shortcode
• Table of content?
• ???

[kaushalmodi]

@chaseadamsio Can you also please add support for tags and publishdate?

#+TAGS: emacs org
#+PUBLISHDATE: 2016-12-14T10:17:43Z

[curiouslychase]

@kaushalmodi PUBLISHDATE should work, did you find that it doesn't? Most standard org headers will parse but I haven't done lists yet (I'm planning on getting that this week). Let me know if you find PUBLISHDATE doesn't work. You can build hugo from this branch after running govendor sync && govendor update github.com/chaseadamsio/goorgeous to get the latest to try it out locally.

[kaushalmodi]

@chaseadamsio Sorry, didn't mean to imply that it did not work. I just went through this thread and missed seeing PUBLISHDATE in examples, that's all.

I don't code in go. But I will give a try building hugo using your PR and goorgeous. Do I then just have a post file with .org instead of .md?

[curiouslychase]

@kaushalmodi yep, that's all you'll need to do. Please let me know if you find any issues in the goorgeous issues as everything I've run into so far has been a minor bug with the dependency and not with hugo.

[kaushalmodi]

I tried .. but didn't work. What am I missing?

km²~/go.apps/:../github.com/chaseadamsio/hugo> git status
On branch orgIntegration
Your branch is up-to-date with 'origin/orgIntegration'.
nothing to commit, working tree clean
km²~/go.apps/:../github.com/chaseadamsio/hugo> govendor sync && govendor update github.com/chaseadamsio/goorgeous
Error: Package "github.com/chaseadamsio/goorgeous" not a go package or not in GOPATH.
km²~/go.apps/:../github.com/chaseadamsio/hugo> pwd
/home/kmodi/go.apps/src/github.com/chaseadamsio/hugo

[curiouslychase]

@kaushalmodi looks like there's probably some stuff we'd have to walk through to get your GOPATH working, can you hit me up on twitter (@chaseadamsio) and I'll help you get a working hugo binary so we don't clutter up the issues and ping a bunch of people with github notifications. :)

[kaushalmodi]

@chaseadamsio I believe I somehow got past that. Looks like go install is working fine as there are no errors. But still it doesn't seem to work.

1. git status gives

   On branch orgIntegration
   Your branch is up-to-date with 'origin/orgIntegration'.
   

2. go install github.com/chaseadamsio/hugo does something.. I get no error messages or affirmative messages. But I do find a new binary in $GOPATH/bin.
3. In my $HUGO_BLOG_ROOT/content/posts/, I create test-org-post.org:

   #+TITLE: My Hugo Blog Post in Org
   #+DATE: 2016-12-14T10:17:43Z
   #+PUBLISHDATE: 2016-12-16T10:17:43Z
   #+DRAFT: true
   
   test
   

4. I have a hugo server running with /path/to/new/hugo server --buildDrafts --buildFuture. I get no errors, but I also don't see that new post titled "My Hugo Blog Post in Org".

[kaushalmodi]

@chaseadamsio Sorry about the noise. Turns out I had to build it from the spf13/hugo dir after add your remote to the same git repo. It works based on the above mentioned test org file. Yay!

So this works:

#+TITLE: My Hugo Blog Post in Org
#+DATE: 2017-02-13
#+PUBLISHDATE: 2017-02-15
#+DRAFT: true

| a | b | c |
| d | e | f |

Org tables are awesome!

I will be writing posts directly in org format now-onwards, and will report issues here, if any,

Thank you for working on this!

[kaushalmodi]

@chaseadamsio I just rebuilt after your yesterday's update to goorgeous. Now I can specify tags and categories too.

I have this in my config.toml:

[taxonomies] # [indexes] is obsolete since v0.11
  category = "categories" # SINGULAR = "PLURAL"

and my theme processes the categories. So this works:

#+TITLE: My Hugo Blog Post in Org
#+DATE: 2017-02-13
#+PUBLISHDATE: 2017-02-15
#+CATEGORIES: abc def
#+DRAFT: true

| a | b | c |
| d | e | f |

[kaushalmodi]

@chaseadamsio I realized that hugo server doesn't rebuild the site when I modify the .org files. The change is detected but the site rebuilding does not happen.

Here is what I see:

Started building sites ...
Built site for language en:
4 of 4 drafts rendered
1 of 1 future rendered
0 expired content
21 regular pages created
19 other pages created
1 non-page files copied
3 paginator pages created
2 tags created
11 categories created
total in 174 ms
Watching for changes in /home/kmodi/hugo/beta/{data,content,layouts,static,themes}
Serving pages from memory
Web Server is available at http://localhost:4444/~kmodi/beta/ (bind address 127.0.0.1)
Press Ctrl+C to stop

Change detected, rebuilding site
2017-02-14 11:21 -0500
Source changed /home/kmodi/hugo/beta/content/posts/test-org-post.org

Above, the site is built when I run hugo server, then when I modify the .org file, I expect the rebuilding to happen. But I just see the "Change detected" message. I need to kill the server and restart to see the changes.

If it matters, I have the post is draft state and I am running hugo server --buildDrafts --buildFuture -p 4444.

---

• I can recreate this issue when editing .md files too. The site doesn't get rebuilt.
• But if I switch to the latest build on spf13/hugo master, hugo server works fine with live edits on .md files.

So may be you need to re-base your branch?

[kaushalmodi]

@chaseadamsio Thanks, live update works now after the rebase.

[bep]

@chaseadamsio if you feel like this is in a "mostly OK" state, we should consider to pull it into master, then ask people on the forum to take it for a spin. Then create new PR(s) for any updates/fixes.

[kaushalmodi]

@bep +1

@chaseadamsio It will be great if this can be merged to the master.

I have only 2 points but those don't seem to be gating factors to this merge:

1. The goorgeous module needs to support more stuff from org spec. But those can go as separate issues on the goorgeous repo.
2. Add a way to use hugo new to create a new .org file. Related -- Add support for org archetypes.

Somewhere it was discussed to add a switch to hugo new that would set the content format. I cannot find it at the moment. So may be have something like hugo new --org?

[bep]

@kaushalmodi you can set markdown=org in the archetype file -- but we should rework the archetype handling for several different reasons, not related to this issue.

[curiouslychase]

@kaushalmodi open the heck out of some issues in https://github.com/chaseadamsio/goorgeous/issues and I'll tackle them as quickly as I can. As far as I've found so far, we're at least matching all the functionality you'd get out of blackfriday, so it's just going to be chipping away at what's left and making sure things like hugo shortcodes are supported.

@bep I think it's ready to go, I started on your list of things from above last night, so maybe we could start a separate issue for any extra things that hugo does that's beyond the blackfriday rendering (that was the main goal for me was to at least have support for org that was in blackfriday for markdown, as well as org headers support so people didn't have to have separate frontmatter that's not a part of the org spec).

If you're comfortable with it and @kaushalmodi feels good about where it is so far (I'm using it on blog posts right now and I'm personally happy with it, but there may be things I don't use that need to be supported), then I give it a 👍 for merging.

[bep]

@chaseadamsio could you:

1. Squash the commits in this PR and write a sensible commit message
2. Make sure the vendor.json is up-to-date

[bep]

@chaseadamsio also, I seem to have committed a slight conflicting change, so this also needs a rebase. It would be good if you could have a look at this in the not so distant future. I would love to get this into Hugo 0.19 (in about a week, I guess), and I would like for this one to linger in master a little bit before the release.

[curiouslychase]

@bep, yep, I'll try and get that worked out tonight. Sorry, I've had family in town & haven't had a chance to do my normal hack time.

[curiouslychase]

@bep updated with squashed commit and resolved your changes.

[github-actions]

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.