Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Export an Org heading to Markdown content in a front-matter variable #272

Closed
xeijin opened this issue Jun 1, 2019 · 22 comments
Closed

Export an Org heading to Markdown content in a front-matter variable #272

xeijin opened this issue Jun 1, 2019 · 22 comments

Comments

@xeijin
Copy link

xeijin commented Jun 1, 2019

I know I'm probably using the incorrect terminology here - but trying to figure out if ox-hugo will be able to do what I want before I dive in, so bear with me:

I want to heavily customise the normal org-mode HTML export, in particular I'd like to:

  1. treat each of the various org elements (titles, headings, drawers, todo keyword etc.) like 'tokens', and place them exactly where I want in a template & style them individually with CSS

  2. create 'new' tokens based on the content of a heading (e.g. the first heading & its associated notes which begins * Attendees:) - again so I can 'place' them within a template

I haven't had much experience with Hugo / web development in general, but would ox-hugo be able to help here? And could you point me to example code that will help me figure out how to do this?

@xeijin xeijin changed the title Question on whether ox-hugo can help with my use-case Question on whether ox-hugo can help with heavily customised org-export html use-case Jun 1, 2019
@kaushalmodi
Copy link
Owner

Hello,

ox-hugo exports to Markdown + a set of Hugo inbuilt + custom meta data if you like.

The Markdown -> HTML conversion is completely done by Hugo (the BlackFriday Markdown parser used by Hugo), so there's little control with the tokenization as you say.

But with the meta data, you have complete control on where to insert those meta data, how to style them using CSS, etc., though all that needs to be done using Hugo templating.

A very simple example is this screenshot that I have on the ox-hugo doc site. Notice how the Org file on the left gets exported to the Hugo front-matter (metadata) + Markdown on the right. The rest (HTML + CSS + ..) is all handled by Hugo.

I hope this intro gives you an idea of what ox-hugo does vs what Hugo does.

@kaushalmodi
Copy link
Owner

This page on Hugo documentation shows few examples of how to use the markdown content front-matter variables in templates: https://gohugo.io/variables/page/#page-level-params

@xeijin
Copy link
Author

xeijin commented Jun 1, 2019

Thanks I'm not sure I'm quite there but that definitely helped.

Here's a question that would help me clarify further as I couldn't figure out from the hugo documentation:

I have an org-mode file as shown before, the 3rd-level headings Attendees and Notes will always be present for each MEETING 2nd-level heading.

#+HUGO_BASE_DIR: ./quickstart
#+HUGO_AUTO_SET_LASTMOD: t
#+TITLE: meeting notes test

* LOG
** MEETING My meeting
*** Attendees
 - [ ] Attendee 1
 - [ ] Attendee 2
*** Notes
**** Note 1
**** Note 2
***** TODO Some action

With ox-hugo this generates the following markdown:

+++
title = "meeting notes test"
lastmod = 2019-06-01T15:49:35+01:00
draft = false
+++

## LOG {#log}


### MEETING My meeting {#meeting-my-meeting}


#### Attendees {#attendees}

-   [ ] Attendee 1
-   [ ] Attendee 2


#### Notes {#notes}

-    Note 1

-    Note 2

    -   <span class="org-todo todo TODO">TODO</span>  Some action

How do I use {#attendees} within a {{ }} token inside of themes/layout/single.html? It causes a syntax error when I try, I'm sure I'm missing something simple here.

@kaushalmodi
Copy link
Owner

kaushalmodi commented Jun 1, 2019

Yes, this is not supported. By this, I mean "converting Org headings to front-matter".

But this can probably work:

#+HUGO_CUSTOM_FRONT_MATTER: :attendees '("attendee 1" "attendee 2")

You can can extract that in the template as .Page.Param "attendees".

Related: https://ox-hugo.scripter.co/doc/custom-front-matter/#list-value-parameters


I believe that extracting headings of Markdown as data was a request in one of the Hugo issues.

@kaushalmodi kaushalmodi changed the title Question on whether ox-hugo can help with heavily customised org-export html use-case Export an Org heading to Markdown content in a front-matter variable Jun 1, 2019
@kaushalmodi
Copy link
Owner

kaushalmodi commented Jun 1, 2019

You can possible even do this:

#+hugo_front_matter_key_replace: author>attendees 
#+author: attendee 1
#+author: attendee 2

Ref:

@xeijin
Copy link
Author

xeijin commented Jun 1, 2019

@kaushalmodi thanks, really helpful example.

I guess my preference is to keep the attendees formatted in org-mode list format (partly because I wrote a script that imports them from an Outlook meeting in this way) and partly because it makes it easy to tick attendees off & add any stragglers.

Bearing that in mind - I guess I would need to write a function that (1) puts the attendees list into the single line required by #+HUGO_CUSTOM_FRONT_MATTER: (2) deletes the existing list of attendees so that this isn't repeated.

Question: is there a hook in ox-hugo that I can use to run said function prior to the content being passed to ox-blackfriday?

@kaushalmodi
Copy link
Owner

is there a hook in ox-hugo that I can use to run said function prior to the content being passed to ox-blackfriday?

No, there isn't one hook, but you can advise the org-hugo--gen-front-matter defun using the :filter-args advice combinator (see C-h i g (elisp) Advice combinators or https://www.gnu.org/software/emacs/manual/html_node/elisp/Advice-combinators.html).

Tweak the input data alist in there and insert in there the attendees list that you parsed using your custom elisp.

You will need to mark the * Attendees subtree with :noexport: tag so that ox-hugo doesn't export it.

Please do share your solution here. If needed, I can make some changes to ox-hugo to make this customization easier for you.

@kaushalmodi
Copy link
Owner

To clarify to what you said earlier:

Bearing that in mind - I guess I would need to write a function that (1) puts the attendees list into the single line required by #+HUGO_CUSTOM_FRONT_MATTER:

  1. You would need to parse the attendees list from the * Attendees subtree (look at the org-elements API) into an elisp list.
  2. Advise org-hugo--gen-front-matter and add `(attendees . ,your-parsed-list) to the data alist arg.

(2) deletes the existing list of attendees so that this isn't repeated.

You wouldn't need to do that .. just tag it :noexport:.

@xeijin
Copy link
Author

xeijin commented Jun 2, 2019

@kaushalmodi one issue I have is that I use the checkbox value ('on 'off or 'nil) to track whether someone actually attended or not.

As an input to org-hugo--gen-front-matter I thought I could use:

(attendees . ("attendee 1" . true) ("attendee 2" . false))

or even better:

(attendees . ((name . "attendee 1") . (attendance . true)) ((name . "attendee 2" . (attendance . false)))

However I can't seem to get these to work? Any thoughts?

@xeijin
Copy link
Author

xeijin commented Jun 2, 2019

Edit: OK, I think I’ve figured out a workaround which is acceptable to me.

Having file header argument (or subtree if the :PROPERTIES: version is used) works exactly as expected, and is flexible enough for my needs:

#+HUGO_CUSTOM_FRONT_MATTER: :attendees (+meeting/get-items-disable-export “Attendees” “noexport”)

Function doing the heavy lift:

(defun +meeting/get-items-disable-export (headline tag)
  "Find the headline exactly matching HEADLINE in buffer, tag with TAG, then find all plain list items under HEADLINE and return as a cons'd list."
  (save-excursion
    (goto-char (org-find-exact-headline-in-buffer headline)) ; jump to heading
    (let* ((hl-as-element)
	   (hl-front-matter))

	; get headline as element
	(unwind-protect 
	    (org-narrow-to-subtree)
	  (setq hl-as-element (org-element-parse-buffer))
	  (widen))

	  ; set tag
	  (let ((hl-tags (org-get-tags))) ; get existing tags
	  (if (member "" hl-tags) ; remove the empty string on headline with no existing tags
	      (setq hl-tags (remove "" hl-tags)))
	  (unless (member tag hl-tags)
	    (push tag hl-tags))
	  (org-set-tags-to hl-tags))

	  ; get items under heading and return as list
	  (setq hl-front-matter (org-element-map hl-as-element 'item ; map over headline's items
				  (lambda (item)
				    (let* ((checkbox (org-element-property :checkbox item)) ; get checkbox value of item
					   (item-text (string-trim (substring-no-properties (org-element-interpret-data (org-element-contents item))))))
				      (cons
				       item-text ; 'org-hugo--gen-front-matter' requires key to be a symbol
				       (cond ; boolean values as strings so that they get converted correctly by 'org-hugo--gen-front-matter'
					((eq checkbox 'on) "true")
					((eq checkbox 'off) "false")
					((eq checkbox 'trans) "false")
					((eq checkbox 'nil) "false"))))))))))

Doing:

C-c C-x C-eHo

in the following org file:

#+HUGO_BASE_DIR: ./quickstart
#+HUGO_CUSTOM_FRONT_MATTER: :attendees (+meeting/get-items-disable-export "Attendees" "noexport")
#+TITLE: meeting notes test

* LOG
** MEETING My meeting
*** Attendees :noexport:
- [X] Attendee 1
- [-] Attendee 2
- [ ] Attendee 3
- Attendee 4
*** Notes
**** Note 1
**** Note 2
***** TODO Some action

Now gives me:

+++
title = "meeting notes test"
draft = false
[attendees]
  Attendee 1 = true
  Attendee 2 = false
  Attendee 3 = false
  Attendee 4 = false
+++

## LOG {#log}


### MEETING My meeting {#meeting-my-meeting}


#### Notes {#notes}

-    Note 1

-    Note 2

    -   <span class="org-todo todo TODO">TODO</span>  Some action

@kaushalmodi
Copy link
Owner

#+HUGO_CUSTOM_FRONT_MATTER: :attendees (+meeting/get-items-disable-export “Attendees” “noexport”)

Wow! I am surprised that that Just Works! I need to read up on plist-get.. I assumed the "value" portion to be just static strings and numbers, but looks like it just evaluates anything present over there!

I need to now document this into ox-hugo and add a test.

Thank you for sharing this.

@kaushalmodi
Copy link
Owner

kaushalmodi commented Jun 3, 2019

@xeijin I played with your elisp snippet. I tried to understand what the role was of the tag input to the defun, but I couldn't understand, and it worked fine even without all the tag-related code.

Regarding the generated TOML:

[attendees]
  Attendee 1 = true
  Attendee 2 = false
  Attendee 3 = false
  Attendee 4 = false

, the "Attendee 1", etc. are invalid TOML keys, so hugo would give you an error. The keys cannot contain spaces AFAIK.

Also, the code calls org-find-exact-headline-in-buffer which does not respect the buffer narrowing done in the subtree based flow. So it will always find the first * Attendees head in the buffer.. even from an another post subtree.

So with those tweaks, I came up with:

(defun xeijin/conv-chkbox-items-to-front-matter (hl)
  "Find the headline exactly matching HL.

Then find all plain list items under HL and return as a
list \\='((checked . (VALa VALb ..)) (not-checked . (VALx VALy
..))).

- The values in \"checked\" cons are the Org list items with
  checkbox in \"on\" state.

- The value in \"not-checked\" cons are the Org list items with
  any other checkbox state, or no checkbox."
  ;; (message "dbg x: pt: %d" (point))
  (let (hl-as-element
        checked not-checked
        ret)
    (save-restriction
      (ignore-errors
        (org-narrow-to-subtree)) ;This will give error when there's no
                                        ;heading above the point, which will
                                        ;be the case for per-file post flow.
      (save-excursion
        (goto-char (point-min))
        ;; (message "dbg y: pt: %d" (point))
        (let (case-fold-search) ;Extracted from `org-find-exact-headline-in-buffer'
          (re-search-forward
	   (format org-complex-heading-regexp-format (regexp-quote hl)) nil :noerror))
        (save-restriction
          (org-narrow-to-subtree) ;Narrow to the `hl' headline
	  (setq hl-as-element (org-element-parse-buffer)))
        ;; (message "dbg: %S" hl-as-element)
        (org-element-map hl-as-element 'item ;Map over headline's items
	  (lambda (item)
	    (let* ((checkbox-state (org-element-property :checkbox item)) ;Get checkbox value of item
		   (item-text (string-trim (substring-no-properties
                                            (org-element-interpret-data
                                             (org-element-contents item))))))
              (cond
               ((eq checkbox-state 'on)
                (push item-text checked))
               (t ;checkbox state in `off' or `trans' state, or if no checkbox present
                (push item-text not-checked))))))
        (setq ret `((checked . ,(nreverse checked))
                    (not-checked . ,(nreverse not-checked))))))
    ;; (message "dbg: ret: %S" ret)
    ret))

which outputs this valid TOML:

[attendees]
  checked = ["Attendee 1"]
  not-checked = ["Attendee 2", "Attendee 3", "Attendee 4"]

Thanks again for this awesome idea to do a function call to dynamically derive the Org keyword value.

kaushalmodi added a commit that referenced this issue Jun 3, 2019
@kaushalmodi
Copy link
Owner

hmm, while this test works locally, it surprisingly fails on Travis only for certain Emacs versions (and I am on Emacs master): https://travis-ci.org/kaushalmodi/ox-hugo/builds/540573276

While I have confirmed this to work locally.. I'll be disabling running of this test on CI.

kaushalmodi added a commit that referenced this issue Jun 3, 2019
- string-trim is not available on all emacs versions
@kaushalmodi
Copy link
Owner

@xeijin OK, the fix was quite straightforward: 434cfff 😄

Let me know if this issue can be closed.

And thanks again!

@kaushalmodi
Copy link
Owner

btw here are the new tests rendered by Hugo:

@xeijin
Copy link
Author

xeijin commented Jun 3, 2019

@kaushalmodi this is excellent, learnt alot from messing with the org-element api over the weekend, so thanks for pointing me in that direction & thank you for the re-factored (improved) function.

@xeijin xeijin closed this as completed Jun 3, 2019
@xeijin
Copy link
Author

xeijin commented Jun 3, 2019

@kaushalmodi a couple of additions I made you may or may not wish to add, I'll re-open but feel free to close if you don't think these are needed.

  • since this is becoming more of a generic 'turn headline with list items into front matter' function, and since Hugo's templating system makes accessing front-matter pretty intuitive, I figured there was no harm in providing the user full range of checkbox states

  • to answer your earlier query, which I missed, the tag input was for setting a tag (typically :no-export:). As you stated earlier, I could simply do this manually, but ...

    • I actually plan to use this in conjunction with with-temp-buffer since I have use-cases where I need the list as front-matter only, and others where I want it to generate like any other org-file
    • may also be useful for folks who use tags other than :noexport: to determine what goes out (though arguably it should perhaps support multiple tags if this is the case) done
    • with that said, I've made adding a tag optional so your tests will still work either way
  • finally, I added a conditional to each of the types to omit it from the final alist if it doesn't exist

(defun xeijin/hl-with-chklst-to-front-matter (hl &rest tags)
  "Find the headline exactly matching HL and, if provided, tag with TAGS then extract plain list items.

The function finds  all plain list items under HL and returns as a
list \\='((checked . (VALa VALb ..)) (unchecked . (VALc VALd
..)) (transient . (VALx VALy ..)) (no-checkbox . (VALz VALq ..))

- The values in \"checked\" cons are the Org list items with
  checkbox in \"on\" [X] state.

- The value in \"unchecked\" cons are the Org list items with
 checkbox in \"unchecked\" [ ] state.

- The value in \"transient\" cons are the Org list items with
  checkbox in \"transient\" [-] checkbox state.

- The value in \"no-checkbox\" cons are the Org list items with
  no checkbox state (i.e. plain list items)."
  
  ;; (message "dbg x: pt: %d" (point))
  (let (hl-as-element
        checked
	unchecked
	transient
	no-checkbox
        ret)
    (save-restriction
      (ignore-errors
        (org-narrow-to-subtree)) ;This will give error when there's no
                                        ;heading above the point, which will
                                        ;be the case for per-file post flow.
      (save-excursion
        (goto-char (point-min))
        ;; (message "dbg y: pt: %d" (point))
        (let (case-fold-search) ;Extracted from `org-find-exact-headline-in-buffer'
          (re-search-forward
	   (format org-complex-heading-regexp-format (regexp-quote hl)) nil :noerror))
        (save-restriction
          (org-narrow-to-subtree) ;Narrow to the `hl' headline
	  (setq hl-as-element (org-element-parse-buffer))
	  
	; set tag(s) if provided
	  (when tags		
	    (let ((hl-tags (org-get-tags))) ; get existing tags
	      ;(if (member "" hl-tags) ; remove the empty string on headline when no existing tags
	 ;	  (setq hl-tags (remove "" hl-tags)))
	     ;(mapc (lambda (x) (push x hl-tags)) tags) ; loop through tags and add each one
	      (nconc hl-tags (nreverse tags)) ; merge new & old tags
	      (setq hl-tags (delq "" (delq nil (delete-dups hl-tags)))) ; remove dupes, nils and empty strings
	      (org-set-tags-to hl-tags)
	      (org-align-all-tags))))
	  
        ;; (message "dbg: %S" hl-as-element)
        (org-element-map hl-as-element 'item ;Map over headline's items
	  (lambda (item)
	    (let* ((checkbox-state (org-element-property :checkbox item)) ;Get checkbox value of item
		   (item-text (org-trim (substring-no-properties
                                            (org-element-interpret-data
                                             (org-element-contents item))))))
              (cond
               ((eq checkbox-state 'on)
                (push item-text checked))
	       ((eq checkbox-state 'off)
	        (push item-text unchecked))
	       ((eq checkbox-state 'trans)
	       (push item-text transient))
               (t ; if no checkbox present
                (push item-text no-checkbox))))))
	
          (setq ret ; ,@: only include in alist when item of that type is present
		  `(,@(when checked `((checked . ,(nreverse checked))))
		    ,@(when unchecked `((unchecked . ,(nreverse unchecked))))
		    ,@(when transient `((transient . ,(nreverse transient))))
		    ,@(when no-checkbox`((no-checkbox . ,(nreverse no-checkbox))))))))
    
    ;; (message "dbg: ret: %S" ret)
    ret))

Example TOML conversions:

*** Attendees
- [X] Attendee 1
- [-] Attendee 2
- [ ] Attendee 3
- Attendee 4

becomes

[attendees]
  checked = ["Attendee 1"]
  unchecked = ["Attendee 3"]
  transient = ["Attendee 2"]
  no-checkbox = ["Attendee 4"]

*** Attendees
- [X] Attendee 1
- [ ] Attendee 2
- [ ] Attendee 3
- Attendee 4

becomes

[attendees]
  checked = ["Attendee 1"]
  unchecked = ["Attendee 2", "Attendee 3"]
  no-checkbox = ["Attendee 4"]

@xeijin xeijin reopened this Jun 3, 2019
@xeijin
Copy link
Author

xeijin commented Jun 3, 2019

Ahhh bugger ... just realised my function is able to set the tags when I run it inside the buffer, but not when I call C-c C-e H o with it inside #+HUGO_CUSTOM_FRONT_MATTER: - any idea why this is? I am thinking perhaps because current-buffer is different when the export is called?

#+HUGO_CUSTOM_FRONT_MATTER: :attendees (xeijin/hl-with-chklst-to-front-matter "Attendees" "noexport" "sometag")

@kaushalmodi
Copy link
Owner

since this is becoming more of a generic 'turn headline with list items into front matter' function, and since Hugo's templating system makes accessing front-matter pretty intuitive, I figured there was no harm in providing the user full range of checkbox states

Yes, the expanded version looks good. The user can then figure out what to do with the different attendees values in the Hugo template itself. As the ox-hugo test is mainly testing the use of elisp in Org keywords, I'll leave that test as is.

to answer your earlier query, which I missed, the tag input was for setting a tag (typically :no-export:).

I see .. I see the role of ox-hugo to touch only the exported files. So I am still refraining from having ox-hugo modify the source Org files. Though, the user is completely free to have Org source modifying elisp.

I am thinking perhaps because current-buffer is different when the export is called?

That's correct. Org mode creates a temp buffer where it expands the Org macros, etc. So the tag insertion is happening there. I believe there's a variable that references back to the original buffer, but you'd need to look into the source code.


At some point, I might add a helper fn in ox-hugo that takes in an elisp function as an arg that outputs plist-friendly data structure.

@xeijin
Copy link
Author

xeijin commented Jun 4, 2019

since this is becoming more of a generic 'turn headline with list items into front matter' function, and since Hugo's templating system makes accessing front-matter pretty intuitive, I figured there was no harm in providing the user full range of checkbox states

Yes, the expanded version looks good. The user can then figure out what to do with the different attendees values in the Hugo template itself. As the ox-hugo test is mainly testing the use of elisp in Org keywords, I'll leave that test as is.

to answer your earlier query, which I missed, the tag input was for setting a tag (typically :no-export:).

I see .. I see the role of ox-hugo to touch only the exported files. So I am still refraining from having ox-hugo modify the source Org files. Though, the user is completely free to have Org source modifying elisp.

Not touching org files makes sense - I guess splitting my function back into tagging vs checklist, then running the tag one on the buffer via with-temp-buffer would actually make more sense, thanks for clarifying your position here.

I am thinking perhaps because current-buffer is different when the export is called?

That's correct. Org mode creates a temp buffer where it expands the Org macros, etc. So the tag insertion is happening there. I believe there's a variable that references back to the original buffer, but you'd need to look into the source code.

At some point, I might add a helper fn in ox-hugo that takes in an elisp function as an arg that outputs plist-friendly data structure.

Thanks again for all your help, learnt a huge amount here.

@xeijin xeijin closed this as completed Jun 4, 2019
@kaushalmodi
Copy link
Owner

Thanks again for all your help, learnt a huge amount here.

You are welcome, but we all are learning here :)

I did not know that we can use elisp forms as Org keyword values.. I am still stoked about that!

Thanks.

@xeijin
Copy link
Author

xeijin commented Jul 14, 2019

@kaushalmodi one other question here, but how would you approach writing tests for something like this? I've never written tests before in general so if you know of a good primer somewhere I'm all ears

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants