New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expand macros *before* the export to get the macro-produced #+hugo_base_dir and such keywords if any #231

Open
FelixBrendel opened this Issue Nov 1, 2018 · 14 comments

Comments

Projects
None yet
2 participants
@FelixBrendel
Copy link

FelixBrendel commented Nov 1, 2018

In my project I do not want to manually set all the HUGO_BASE_DIR properties, so I wrote a simple org file, that when #+include'd inserts the for my project appropriate values for HUGO_BASE_DIR and HUGO_SECTION independent from the files location. The included file contains these lines.

#+macro: hugosection (eval (concat "#+hugo_section: "  (directory-file-name (file-relative-name default-directory (concat (projectile-project-root) "org")))))
#+macro: hugobasedir (eval (concat "#+hugo_base_dir: " (directory-file-name (file-relative-name (concat (projectile-project-root) "hugo") default-directory))))

{{{hugosection}}}
{{{hugobasedir}}}

I confirmed, that when exported to org, my documents have the correct paths set. However it seems that ox-hugo just checks if these properties are there in clear text before the macros get expanded.

A file containing only my include

#+include: ../preload.org

will be transformed to

# Created 2018-11-01 Do 01:12
#+TITLE: 
#+AUTHOR: Felix Brendel
#+MACRO: hugosection (eval (concat "#+hugo_section: "  (directory-file-name (file-relative-name default-directory (concat (projectile-project-root) "org")))))
#+MACRO: hugobasedir (eval (concat "#+hugo_base_dir: " (directory-file-name (file-relative-name (concat (projectile-project-root) "hugo") default-directory))))

#+HUGO_SECTION: foldername
#+HUGO_BASE_DIR: ../../hugo

By the macro expansion (the paths are correct).

By temporarily commenting out the error I verified that the export would indeed work without any trouble.

Is there another way to turn off the error, or to delay the check somehow to after the macro expansion phase?

@kaushalmodi kaushalmodi closed this Nov 1, 2018

@kaushalmodi kaushalmodi reopened this Nov 1, 2018

@kaushalmodi

This comment has been minimized.

Copy link
Owner

kaushalmodi commented Nov 1, 2018

Hello,

That's an interesting way to do what you want, but I believe there might be easier ways to do the same or similar.

By definition of Org Macros (See (org) Macro Replacement):

Macros replace text snippets during export.

We are use the #+hugo_.. keywords before we start the export, and then the Org macros get expanded.

I confirmed, that when exported to org, my documents have the correct paths set.

That "works" for ox-org export because that export is literally expanding that macro and replacing that whatever text that generated.. it doesn't further use the replacement in any way.

Proof is in the pudding, so here's a modified version of your example that "works" for ox-hugo too (after I removed "#+" from the macro expansions):

#+hugo_base_dir: ../.
#+macro: hugosection (eval (concat "hugo_section: "  (directory-file-name (file-relative-name default-directory (concat (projectile-project-root) "org")))))
#+macro: hugobasedir (eval (concat "hugo_base_dir: " (directory-file-name (file-relative-name (concat (projectile-project-root) "hugo") default-directory))))

#+title: Issue 231

{{{hugosection}}}
{{{hugobasedir}}}

That exports to:

+++
title = "Issue 231"
author = ["Kaushal Modi"]
draft = false
+++

hugo\_section: ../test/site/content-org
hugo\_base\_dir: ../../../hugo

Note that I have removed "#+" from the expanded string because "#+" prepended lines will not be exported by ox-hugo (or any other Org exporter for that matter, except ox-org of course). Try exporting your example using C-c C-e t A (ascii export to buffer). The "#+" expansions of the macros won't be found.

The reason you see in ox-org is, well, because you are exporting an Org file to an Org file, and you don't want to lose any information, even the Org keywords.


In my project I do not want to manually set all the HUGO_BASE_DIR properties, so I wrote a simple org file, that when #+include'd ..

Usually adding those two keywords is not much of a pain. It's usually something like:

#+hugo_base_dir: ../.
#+hugo_section: notes
  1. If you are using per-file flow, and you need to add it to each and every post file, you can use the #+setupfile keyword. Save the above to a foo.org file and add #+setupfile: /path/to/that/foo.org at the top of each post file.
  2. Even if you are using per-subtree flow, above would apply. Or I'd just manually type the above.
  3. Finally, there are tools like yasnippet, et al that can insert the above automatically for you.
  4. Or transform those Org macros to a little interactive elisp function that you can call to auto-insert the above.

By temporarily commenting out the error I verified that the export would indeed work without any trouble.

I wouldn't advise you to do that. The error is there for a reason.

The parsed base-dir is used here:

ox-hugo/ox-hugo.el

Lines 1258 to 1260 in 03c146e

(pub-dir (let ((dir (concat base-dir content-dir section-path bundle-dir)))
(make-directory dir :parents) ;Create the directory if it does not exist
dir)))

If you comment out the error, the base-dir will be nil, the concat won't complain about nil base-dir, and make-directory will create a new directory at a possibly unintended location.

Also the section will default to "posts".

Basically, both base dir and section get parsed as nil. So don't comment out any of the errors, and do one of the suggested solutions above.


PS: Your Org macro definitions don't look right. The section name is a dir name, and the base dir is not necessarily the project root, but rather the dir containing the config.toml (or yaml ..).

@FelixBrendel

This comment has been minimized.

Copy link
Author

FelixBrendel commented Nov 1, 2018

Thank you very much for your elaborate answer!

Note that I have removed "#+" from the expanded string because "#+" prepended lines will not be exported by ox-hugo (or any other Org exporter for that matter, except ox-org of course). Try exporting your example using C-c C-e t A (ascii export to buffer). The "#+" expansions of the macros won't be found.

I think this is not quite correct.. I have already successfully used macros to generate lines starting with "#+". For example when writing my thesis I used a todo macro, which would output the given reminder in red and add one to a latex counter.

#+macro: todo (eval "\n#+latex:\\textcolor{red}{\\textbf{todo($1)}} \\addtocounter{todoCounter}{1}")

I then exported directly tex and compiled with latexmk. So unless this an exception, it works in general to macroexpand "#+" lines, which for me is one of the reason org is great.

To come back to this example, I know it is just entering the correct path, but I do not want to care about it for every file, also the path will not be the same for every file because the files will be in different subfolders, of varying nested depth, that's why it was tempting to write a macro which just finds the correct path.

PS: Your Org macro definitions don't look right. The section name is a dir name,

Yes, like this I can structure my org folders as I want and every org file will be exported to a section corresponding to the folder the org file lives in.

Again, my overall goal is to have a really flexible folder structure where all my org files live (so no common hugo_base_dir between them) and then have them all export into the same folder structure inside the "content" folder.

For that I wrote some lisp to walk all the files and export them, and now first export them to org before to hugo, which works as a workaround

(progn
  (require 'ox-hugo)
  (mapcar
   (lambda (file)
     (unless (string= "preload" (file-name-sans-extension (file-name-nondirectory file)))
       (find-file-other-window file)
       (org-org-export-as-org)
       (let ((tmp-file (concat (file-name-sans-extension (file-name-nondirectory file)) ".orgexpanded")))
         (write-file tmp-file)
         (org-hugo-export-to-md)
         (delete-file tmp-file)
         (kill-buffer tmp-file))))

   (directory-files-recursively default-directory ".org$"))
  t)

But regardless I want to stress again (I checked again right now just to be 100% sure), that when disabling the error, the exported directory is correct. Which has to mean, that the macro expansion does happen during the export process.

Thank you for your help!

@kaushalmodi

This comment has been minimized.

Copy link
Owner

kaushalmodi commented Nov 1, 2018

Hmm, I spent quite some time on this yesterday, but couldn't make it work. Let me find more time this week.

Thanks for the Org macro example for ox-latex export. I'll try that out today to see if it works.

In case that doesn't work too, is there a defcustom you have set that enables this 2-pass keyword evaluation?

To make this work, Org is first evaluating all the #+macro keywords, and then it is evaluating the generated "#+.." keywords, and then it finally exports.

If I can reproduce your example for ox-latex export, then there are higher chances that I can make that happen for ox-hugo too.

@kaushalmodi

This comment has been minimized.

Copy link
Owner

kaushalmodi commented Nov 1, 2018

But regardless I want to stress again (I checked again right now just to be 100% sure), that when disabling the error, the exported directory is correct.

That part, I verified few times too. If I used your macro and commented out that error, the calculated value of the base dir always remained nil.

So I don't understand why just commenting out the error works for you.

Are you sure you haven't done some customization to the export flow?

@FelixBrendel

This comment has been minimized.

Copy link
Author

FelixBrendel commented Nov 1, 2018

Even I made sure, by deleting the md file and manually exporting it again with C-c C-e H h and checking the resulting file, I am now in a position where I tried to create a minimal working example but it turns out, it does not work. That is why I am now really wondering how it worked before since as I said I checked it two times! About the latex macro: I am right now creating a minimal working example for that to show you!

@FelixBrendel

This comment has been minimized.

Copy link
Author

FelixBrendel commented Nov 1, 2018

Okay the latex example definitely works:
The minimal org file for that is:

#+latex_header:\newcounter{todoCounter}
#+latex_header:\setcounter{todoCounter}{0}
#+latex_header:\AtEndDocument{\ifnum\value{todoCounter}>0\typeout{! Todos There are \thetodoCounter \space todos in this document left.}\else\typeout{No todos found.}\fi}
#+latex_header: \usepackage{xcolor}

#+macro: todo (eval "\n#+latex:\\textcolor{red}{\\textbf{todo($1)}} \\addtocounter{todoCounter}{1}")


* Content
With todos
{{{todo(aslkjd)}}}

The resulting tex file is:

% Created 2018-11-01 Do 13:38
% Intended LaTeX compiler: pdflatex
\documentclass[11pt]{article}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{fixltx2e}
\usepackage{graphicx}
\usepackage{grffile}
\usepackage{longtable}
\usepackage{wrapfig}
\usepackage{rotating}
\usepackage[normalem]{ulem}
\usepackage{amsmath}
\usepackage{textcomp}
\usepackage{amssymb}
\usepackage{capt-of}
\newcounter{todoCounter}
\setcounter{todoCounter}{0}
\AtEndDocument{\ifnum\value{todoCounter}>0\typeout{! Todos There are \thetodoCounter \space todos in this document left.}\else\typeout{No todos found.}\fi}
\usepackage{xcolor}
\author{Felix Brendel}
\date{\today}
\title{}
\begin{document}

\tableofcontents



\section{Content}
\label{sec:org6b05539}
With todos

\textcolor{red}{\textbf{todo(aslkjd)}} \addtocounter{todoCounter}{1}
\end{document}

The special thing about the todo macro is that it increments a latex variable, and when compiling, at then end, there were todos, it will print out how many todos are found in the document, so I wont have any todos accidentally left in the document. (Red line on the right)

image

The resulting pdf:
image

@kaushalmodi

This comment has been minimized.

Copy link
Owner

kaushalmodi commented Nov 1, 2018

@FelixBrendel I confirm that your latex example works; just that I needed to modify the eval form to below to make it work on Org master branch (upcoming Org 9.2):

#+macro: todo (eval (concat "\n#+latex:\\textcolor{red}{\\textbf{todo(" $1 ")}} \\addtocounter{todoCounter}{1}"))

Now the eval form on Org macros accepts only strings, and the Org macro arguments are strings by default.

Now I need to investigate why this works but not the #+hugo_* keywords. My first guess would be that those particular hugo switches need to be evaluated before the actual export begins, while the #+latex switch is evaluated during an export.

@kaushalmodi

This comment has been minimized.

Copy link
Owner

kaushalmodi commented Nov 1, 2018

OK, it was easy to prove that theory. For example, #+hugo: more is parsed during ox-hugo export here. So the below example works:

#+hugo_base_dir: ../.

#+macro: more (eval (concat "\n\n#+hugo: " "more"))

#+title: Issue 231 - Hugo more

{{{more}}}

export to (on doing C-c C-e H H):

+++
title = "Issue 231 - Hugo more"
author = ["Kaushal Modi"]
draft = false
+++

<!--more-->

Above is a trivial example that basically expands {{{more}}} to #+hugo: more.

So, we now need to figure out how to make Org evaluate the pre-export setup keywords like #+hugo_base_dir, and #+hugo_section.

@kaushalmodi

This comment has been minimized.

Copy link
Owner

kaushalmodi commented Nov 1, 2018

So the macro expansion seems to be happening in org-export-as in ox.el. Seems like we will be fighting the Org code as we are messing with macro expansion outside of the actual export.

I still like the idea, but don't have enough time to investigate this further. I'll accept the PR if you get to the bottom of this.

@kaushalmodi kaushalmodi changed the title HUGO_BASE_DIR property cannot be set via macros Expand macros *before* the export to get the macro-produced #+hugo_base_dir and such keywords if any Nov 1, 2018

@kaushalmodi

This comment has been minimized.

Copy link
Owner

kaushalmodi commented Nov 1, 2018

Bit of explanation:

image

@FelixBrendel

This comment has been minimized.

Copy link
Author

FelixBrendel commented Nov 1, 2018

Thank you for your hard work and being an excellent package author / maintainer! I just want to note that this is not urgent for me. I think for my case it would be easier to generate a hugo.org file programatically that includes all my org files in the right order, with the right weight and in the right sections, so that I won't have to write all the properties manually, which would be error prone. And then just ox-hugo export all subtrees to the different md files.

@kaushalmodi

This comment has been minimized.

Copy link
Owner

kaushalmodi commented Nov 1, 2018

I think for my case it would be easier to generate a hugo.org file programatically that includes all my org files in the right order, with the right weight and in the right sections,

Have you considered just maintaining a single (or few) Org files with subtrees.

so that I won't have to write all the properties manually, which would be error prone.

What kind of properties do you think would be error prone when writing. Subtree based approach will allow you to take a huge advantage of property and tag inheritance. You don't even have to set the weights; they can be set automatically.

In any case, it would be great if you can share your Org source whenever you are done using whatever approach you choose :)

Thank you for your hard work and being an excellent package author / maintainer!

You are welcome!

@FelixBrendel

This comment has been minimized.

Copy link
Author

FelixBrendel commented Nov 1, 2018

I would love to ask you some questions about the way one "should" set up a project like this and also ask you about my specific case, I think this comment section however would not be the right place for that. Do you have any preferred way we could exchange some thoughts?

@kaushalmodi

This comment has been minimized.

Copy link
Owner

kaushalmodi commented Nov 1, 2018

Do you have any preferred way we could exchange some thoughts?

You can find me at https://gitter.im/KaushalModi/Lobby.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment