Extension to treat first heading level as title? #5615

DanielSWolf · 2019-06-23T19:16:45Z

I'm using Pandoc 2.7.3 to convert Markdown to AsciiDoc. In the output file, the level of every heading is 1 deeper than expected.

README.md:

# Title

## Level 2

Invocation:

pandoc -o README.adoc README.md

Output (README.adoc):

== Title

=== Level 2

Expected output:

= Title

== Level 2

The text was updated successfully, but these errors were encountered:

jgm · 2019-06-23T23:12:59Z

The = is reserved for the document title, provided in metadata.
This is standard in asciidoc; see e.g. https://raw.githubusercontent.com/asciidoctor/asciidoctor.org/master/docs/asciidoc-syntax-quick-reference.adoc as an example.

jgm · 2019-06-23T23:13:39Z

Use this form:

---
title: Title
...

# First section

DanielSWolf · 2019-06-24T10:35:06Z

I'm not sure that I got my problem across. I realize that in AsciiDoc, = is the document title. The thing is that Markdown has no comparable syntax for defining the document title. As far as I know, the syntax you mention in your previous comment isn't standard Markdown. So in practice, # is often used in Markdown documents for the document title, then ##, ###, etc. are used for headings -- just like in AsciiDoc.

I have a large number of Markdown files that use this approach and that I'd like to convert to AsciiDoc. So I need to convert # to =, ## to ==, and so on. Is there a way to achieve this?

mb21 · 2019-06-24T11:47:15Z

Well, pandoc uses pandoc flavoured markdown...

I was thinking you can could write a lua-filter to decrement all header levels by one, similar to this example: https://pandoc.org/lua-filters.html#modifying-pandocs-manual.txt-for-man-pages, but this causes the current pandoc version to crash on Prelude.init: empty list somewhere in the asciidoc writer.

You could change this line and compile pandoc yourself...

mb21 · 2019-06-24T12:30:18Z

Specifically, the hierarchicalizeWithIds function in Pandoc.Shared breaks...

mb21 · 2019-06-24T12:42:36Z

Or write a filter that decrements all heading levels by one, except for the heading 1, which should be set as the document meta data title.

tarleb · 2019-06-24T13:14:11Z

Relevant StackOverflow question and answer: Pandoc: set document title to first title

agusmba · 2019-06-24T14:26:03Z

@tarleb you're a lua wizard! I just grabbed another one of yours for numbered-chapter-reference ~~(not enough karma for my votes to show yet)~~

jgm · 2019-06-25T20:23:45Z

Maybe we need a markdown extension that treats a unique level-one header as the metadata title, and promotes all other headers. Something like +top_heading_as_title.

mb21 · 2019-06-26T08:13:42Z

That would solve it for markdown.. but what about reading other formats like HTML?

jgm · 2019-06-26T21:46:36Z

We could make the extension affect HTML as well. I don't know if there are other formats that use this convention for indicating titles.

mb21 · 2019-08-30T11:10:54Z

This issue is more vexing to me than expected...

It seems to me now that what we need is an extension that treats a unique level-one header as the metadata title, and does nothing to the other headers. Let me explain...

This markdown:

---
title: Title
---

## Level 2

converted with pandoc -s, results in this HTML structure:

<head>
  <title>Title</title>
</head>
<body>
  <header>
    <h1 class="title">Title</h1>
  </header>
  <h2>Level 2</h2>
</body>

which is usually what you want: one h1, one h2.

Now to the asciidoc and html readers:

If you're converting from pandoc markdown (same md as above):

```
% pandoc -t asciidoc -s
---
title: foo
---

## bar
^D
= foo

== bar
```

If you're converting from markdown with a different header convention (OP's md):

```
% pandoc -f markdown+top_heading_as_title -t asciidoc -s
# foo

## bar
^D
= foo

== bar
```


Similarly from HTML; currently we have:

```
% pandoc -f html -t markdown --atx-headers -s
<html>
<head>
  <title>foo</title>
</head>
<body>
  <h1>foo2</h1>
  <h2>bar</h2>
</body>
</html>
^D
---
title: foo
---

# foo2

## bar
```

But what you usually want when converting a website to pandoc markdown is:

```
% pandoc -f html+top_heading_as_title -t markdown --atx-headers -s
<html>
<head>
  <title>foo</title>
</head>
<body>
  <h1>foo2</h1>
  <h2>bar</h2>
</body>
</html>
^D
---
title: foo2
---

## bar
```

Does that make sense?

jgm · 2019-08-30T16:02:22Z

it makes sense, but that doesn't mean we should parse h2 -> Header 2 in this case.

Converting HTML -> LaTeX, with this style of HTML, you'd generally want the h2's to convert to \section, not \subsection. So they'd need to be Header 1 in the AST. The same is true for many other formats pandoc supports.

One could try to address this at the HTML writer level. When the special extension or option is set, Header 1 renders as h2.

So, to summarize:

with -f html+top_heading_as_title, h1 goes to the metadata title and h2 goes to a Header 1.
with -t html+top_heading_as_title, metadata title goes to h1 and Header 1 goes to h2.

mb21 · 2019-08-31T07:42:08Z

Ah, considering LaTeX output is indeed interesting.

I had the impression now that the recommend md for html export is the following (otherwise you get two <h1> with -s, which is usually not recommended):

---
title: foo
---

## bar

But you're saying that for LaTeX export you'd usually want the following?

---
title: foo
---

# bar

jgm · 2019-08-31T16:35:28Z

But you're saying that for LaTeX export you'd usually want the following?

Yes, for LaTeX and most other formats. With the ## you'd get subsections numbered 0.1, 0.2, etc. I also produce HTML this way (it's the default for pandoc), but I understand this has drawbacks.

Relevant old issue: #686.

mb21 · 2019-09-08T10:55:26Z

I've been thinking about this some more and made peace with the fact that depending on what you're doing you'll always want to adjust your heading levels. Might depend on the output format you're converting to, or more importantly on where in any existing hierarchy your piece of text will fit into: say an existing website that uses <h1> for its logo text, or maybe you're writing a book and converting individual md files one by one to HTML pages (each should have an h1), but you concatenate all your md files when going to LaTeX.

So there will always be cases where you should simply make use of the --base-header-level option. Two things:

I'm not sure --base-header-level is the most intuitive name. It kind of leaves open what happens to all headings that are not at the 'base level'. What about something like --shift-heading-levels-by with the default value being 0?
Wouldn't it be nice if the option could also take negative values? If you would do --shift-heading-levels-by=-1 the first heading with level 1 would be set as the metadata title, and we wouldn't need yet another option/extension. I suppose other headings could be dropped, as well as headings that end up having a level <= 0.

jgm · 2019-09-09T03:52:47Z

Interesting idea. But isn't it a bit odd if

shift by -1 makes a level-1 heading the metadata title

unless also

shift by +1 makes metadata title a level-1 heading ?

This latter would definitely be a change to how --base-header-level=2 currently works.

EDIT: Even so I'm pretty positively disposed to this idea. I believe people have requested negative heading level shifts before. (See #4342)

mb21 · 2019-09-09T07:07:54Z

shift by +1 makes metadata title a level-1 heading

yes, I think that would be useful in some rare cases as well.

I suppose --base-header-level should be deprecated then...

mb21 · 2019-09-11T07:40:33Z

🎉

brainchild0 · 2019-09-20T02:50:28Z

I think it makes sense that shifting between metadata and level-1 heading occurs in both directions. I don't find a compelling counter-example. But if the effect applies to all inputs after concatenation, then can the user not provide a document title that doesn't get demoted to a mere header?

Sometimes users split a book into files. Or a book may constitute a compilation of articles from sources written originally for standalone publication.

For example, consider inputs:

---
title:  Book Title
---

---
title: Beginning
---

In the beginning...

---
title: Ending
---

At the end...

The intention might be to represent:

---
title:  Book Title
---

# Beginning

In the beginning...

# Ending

At the end...

Could the global metadata input be protected from demotion? Could shifting be selected at the granularity of individual inputs, and applied before concatenation?

And would a title be handled differently if coming from the metadata given on the command line versus a YAML source? I wouldn't suggest a solution in which giving a title on the command line is the only way to protect it from demotion.

jgm · 2019-09-20T04:58:55Z

@brainchild0 see --file-scope.

brainchild0 · 2019-09-20T05:42:26Z

Would this switch guard any single input file from transformations that would apply to others? I am not seeing any indication of such in the manual.

jgm · 2019-09-20T14:29:41Z

I don't understand the question. (But I recommend you experiment to find out.)

brainchild0 · 2019-09-20T19:25:44Z

Following are the best experiments I can do currently:

Looking at my first post in the issue, notice a sequence of three examples of file contents following the line beginning with "For example". I place them in a.md, b.md, and c.md. The example following "The intention" I place in x.md.

Now I try:

$ pandoc x.md 
<h1 id="beginning">Beginning</h1>
<p>In the beginning…</p>
<h1 id="ending">Ending</h1>
<p>At the end…</p>

The files a.md, b.md, and c.md represent the idea that file x.md is being decomposed into parts. Since the latter two files represent chapters in the form of stanadalone documents, the level-1 headings are represented as the document titles.

The idea expressed in that post was that it might be useful if I could run a command using these three files, but is equivalent to pandoc x.md. This involves shifting b.md and c.md to the right one level, but "guarding" a.md.

With the current options, I think it is impossible. The closest approximation would be

$ pandoc a.md b.md c.md --shift-heading=1
<h1>Book Title</h1>
<p>In the beginning…</p>
<p>At the end…</p>

Actually, the result currently is that the chapter titles are dropped, because only one title may be recognized for the document.

With --file-scope there is no difference:

$ pandoc a.md b.md c.md  --shift-heading=1  --file-scope
<h1>Book Title</h1>
<p>In the beginning…</p>
<p>At the end…</p>

What would be needed is a way to shift the contents of b.md and c.md to right one level, such that the titles are demoted to level-1 headings, while a.md is "guarded", providing the actual title of the document.

At first it may seem like an unusual case, but I think probably not so. It seems that currently any positive shift value prevents the input stream from giving any data that is used for the document title of the output.

jgm · 2019-09-20T22:02:51Z

What would be needed is a way to shift the contents of b.md and c.md to right one level, such that the titles are demoted to level-1 headings, while a.md is "guarded", providing the actual title of the document.

Correct, that can't currently be done.
It would be desirable to make this sort of thing possible using a combination of --file-scope and --metadata-file. But that combination doesn't seem to work the way I'd expect.

% cat m.yaml
title: my real title
% cat a.md
---
title: hi
...

# ok
% pandoc --file-scope --metadata-file m.yaml a.md -t native -s
Pandoc (Meta {unMeta = fromList [("title",MetaInlines [Str "hi"])]})
[Header 1 ("ok",[],[]) [Str "ok"]]
% pandoc --file-scope --metadata-file m.yaml a.md -t native -s --shift-heading=1
Pandoc (Meta {unMeta = fromList []})
[Header 1 ("",[],[]) [Str "hi"]
,Header 2 ("ok",[],[]) [Str "ok"]]

I'd find it more intuitive (and useful) if the heading-shift transformation was done before the metadata was integrated, so the metadata from m.yaml isn't clobbered. However, this should be a new issue.

jgm · 2019-09-20T22:06:21Z

Actually on reflection, I'm not so sure about this.

brainchild0 · 2019-09-20T23:26:07Z

I'd find it more intuitive (and useful) if the heading-shift transformation was done before the metadata was integrated, so the metadata from m.yaml isn't clobbered. However, this should be a new issue.

It's not obvious how to create a new issue that captures the immediate concerns without including the history.

The sequence of processing would seem to be close to the following:

Collect each file not including the one (or ones) containing the global metadata.
For each file collected as such:
1. Interpret it, including the metadata within it.
2. Apply any appropriate shift, including, in the case of a positive shift, changing the title from the metadata into a heading.
3. (Discard other metadata... I assume... or not?)
Concatenate the results.
Apply the global metadata.

Which parts of this discussion, if any, would you want moved to a new issue, and which would you be less open to seriously considering at the current moment?

brainchild0 · 2019-09-24T05:15:21Z

Also, not sure about a compelling use case, but if a left shift squashes several header levels into one, then the original level of each affected header in principle can be preserved in ancillary data, like XHTML data- elements or class tags (e.g. <h1 data-original-level="-1">).

jgm · 2019-12-03T16:47:34Z

See #5957 for an unintended consequence of this change.

jgm · 2019-12-05T17:45:30Z

I'm going to roll back:

shift by +1 makes metadata title a level-1 heading

This breaks some workflows that used to be supported with --base-heading-level (see #5957).

Also: suppose you want to render a document with both latex and html. IT would be natural to use level-1 headings. But in the HTML version you might want level-2 headings, since the title will be rendered as level-1. So you'd want to shift heading levels, without depopulating the title in metadata.

With positive heading shifts, starting in 2.8 this option caused metadata titles to be removed and changed to regular headings. This behavior is incompatible with the old behavior of `--base-header-level` and breaks old workflows, so with this commit we are rolling back this change. Now, there is an asymmetry in positive and negative heading level shifts: + With positive shifts, the metadata title stays the same and does not get changed to a heading in the body. + With negative shifts, a heading can be converted into the metadata title. I think this is a desirable combination of features, despite the asymmetry. One might, e.g., want to have a document with level-1 section headigs, but render it to HTML with level-2 headings, retaining the metadata title (which pandoc will render as a level-1 heading with the default template). Closes #5957. Revises #5615.

With positive heading shifts, starting in 2.8 this option caused metadata titles to be removed and changed to regular headings. This behavior is incompatible with the old behavior of `--base-header-level` and breaks old workflows, so with this commit we are rolling back this change. Now, there is an asymmetry in positive and negative heading level shifts: + With positive shifts, the metadata title stays the same and does not get changed to a heading in the body. + With negative shifts, a heading can be converted into the metadata title. I think this is a desirable combination of features, despite the asymmetry. One might, e.g., want to have a document with level-1 section headigs, but render it to HTML with level-2 headings, retaining the metadata title (which pandoc will render as a level-1 heading with the default template). Closes jgm#5957. Revises jgm#5615.

rauschma · 2021-08-19T13:24:23Z

I use --shift-heading-level-by=-1 here:

# Title of document
## First section
## Second section

I love that the title of the document is now specified via # – it’s more “WYSIWYG” in that the output looks closer to common Markdown previews (Visual Studio Code etc.).

The title is displayed via <h1>. Alas, the sections are also <h1>, which seems wrong. If I wrote the output HTML by hand, section titles would be <h2>.

jgm · 2021-08-19T16:32:32Z

Given the recommendation only to use one h1 element per page, it would make a certain amount of sense to map a pandoc Header 1 to HTML <h2>, Header 2 to <h3>, etc., reserving <h1> for the title if any.
You could raise this for discussion on pandoc-discuss. It would be a pretty big change in behavior.

the-solipsist · 2023-09-07T18:52:44Z

@rauschma's use case (which happens to be mine) can be solved by using this lua filter (adapted with a single changed line from one that @tarleb posted in 2019). This filter will make all # headings into <title> headings with <h1>, but will keep all lower-level headings (##, etc.) as they are.

local title

-- Set title from level 1 header, or
-- discard level 1 header if title is already set.
function make_header1_title (header)

  if header.level >= 2 then
    return header
  end

  if not title then
    title = header.content
    return {}
  end

  local msg = '[WARNING] title already set; discarding header "%s"\n'
  io.stderr:write(msg:format(pandoc.utils.stringify(header)))
  return {}
end

return {
  {Meta = function (meta) title = meta.title end}, -- init title
  {Header = make_header1_title},
  {Meta = function (meta) meta.title = title; return meta end}, -- set title
}

jgm closed this as completed Jun 23, 2019

jgm reopened this Jun 25, 2019

jgm changed the title ~~Heading levels are changed~~ Extension to treat first heading level as title? Jul 20, 2019

mb21 added the enhancement label Jul 24, 2019

jgm mentioned this issue Aug 9, 2019

Feature Request: TOC Stop Depth and Start Depth #5676

Closed

mb21 mentioned this issue Aug 30, 2019

New markdown to asciidoc tests #5716

Open

mb21 mentioned this issue Sep 5, 2019

Issue of --numbered-sections used with --base-header-level=2: produces 0.x.y #5071

Closed

jgm added this to the 2.8 milestone Sep 9, 2019

jgm closed this as completed in 88dc6fa Sep 11, 2019

alerque mentioned this issue Jan 18, 2020

Render === titles similar to # titles Zettlr/Zettlr#478

Closed

ysard mentioned this issue Feb 2, 2022

About SEO report and h1 tags pelican-plugins/seo#55

Open

badlydrawnrob mentioned this issue Jan 13, 2024

Pandoc output garbled (unicode characters etc) badlydrawnrob/anki#82

Closed

leodarrigade mentioned this issue May 3, 2024

Use of # header level 1 as notebook title ? jupyter/nbconvert#1901

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Extension to treat first heading level as title? #5615

Extension to treat first heading level as title? #5615

DanielSWolf commented Jun 23, 2019 •

edited

jgm commented Jun 23, 2019

jgm commented Jun 23, 2019

DanielSWolf commented Jun 24, 2019

mb21 commented Jun 24, 2019

mb21 commented Jun 24, 2019

mb21 commented Jun 24, 2019

tarleb commented Jun 24, 2019

agusmba commented Jun 24, 2019 •

edited

jgm commented Jun 25, 2019

mb21 commented Jun 26, 2019

jgm commented Jun 26, 2019

mb21 commented Aug 30, 2019

jgm commented Aug 30, 2019

mb21 commented Aug 31, 2019

jgm commented Aug 31, 2019

mb21 commented Sep 8, 2019 •

edited

jgm commented Sep 9, 2019 •

edited

mb21 commented Sep 9, 2019

mb21 commented Sep 11, 2019

brainchild0 commented Sep 20, 2019 •

edited

jgm commented Sep 20, 2019

brainchild0 commented Sep 20, 2019

jgm commented Sep 20, 2019

brainchild0 commented Sep 20, 2019

jgm commented Sep 20, 2019

jgm commented Sep 20, 2019

brainchild0 commented Sep 20, 2019 •

edited

brainchild0 commented Sep 24, 2019

jgm commented Dec 3, 2019 •

edited

jgm commented Dec 5, 2019

rauschma commented Aug 19, 2021 •

edited

jgm commented Aug 19, 2021

the-solipsist commented Sep 7, 2023 •

edited

Extension to treat first heading level as title? #5615

Extension to treat first heading level as title? #5615

Comments

DanielSWolf commented Jun 23, 2019 • edited

jgm commented Jun 23, 2019

jgm commented Jun 23, 2019

DanielSWolf commented Jun 24, 2019

mb21 commented Jun 24, 2019

mb21 commented Jun 24, 2019

mb21 commented Jun 24, 2019

tarleb commented Jun 24, 2019

agusmba commented Jun 24, 2019 • edited

jgm commented Jun 25, 2019

mb21 commented Jun 26, 2019

jgm commented Jun 26, 2019

mb21 commented Aug 30, 2019

jgm commented Aug 30, 2019

mb21 commented Aug 31, 2019

jgm commented Aug 31, 2019

mb21 commented Sep 8, 2019 • edited

jgm commented Sep 9, 2019 • edited

mb21 commented Sep 9, 2019

mb21 commented Sep 11, 2019

brainchild0 commented Sep 20, 2019 • edited

jgm commented Sep 20, 2019

brainchild0 commented Sep 20, 2019

jgm commented Sep 20, 2019

brainchild0 commented Sep 20, 2019

jgm commented Sep 20, 2019

jgm commented Sep 20, 2019

brainchild0 commented Sep 20, 2019 • edited

brainchild0 commented Sep 24, 2019

jgm commented Dec 3, 2019 • edited

jgm commented Dec 5, 2019

rauschma commented Aug 19, 2021 • edited

jgm commented Aug 19, 2021

the-solipsist commented Sep 7, 2023 • edited

DanielSWolf commented Jun 23, 2019 •

edited

agusmba commented Jun 24, 2019 •

edited

mb21 commented Sep 8, 2019 •

edited

jgm commented Sep 9, 2019 •

edited

brainchild0 commented Sep 20, 2019 •

edited

brainchild0 commented Sep 20, 2019 •

edited

jgm commented Dec 3, 2019 •

edited

rauschma commented Aug 19, 2021 •

edited

the-solipsist commented Sep 7, 2023 •

edited