Chops off trailing .Content in one multioutput setup #4706

kaushalmodi · 2018-05-07T16:44:59Z

Hello,

I realized that Hugo 0.40.1 introduced a serious regression.

It chops off some of the trailing .Content!

This issue is not seen in hugo server.
It is only seen when the site is built to public/ using the hugo command.

Bug example

~~Live example of this bug~~ (not live any more) (Search for "Resources" to get to the point of truncated .Content) | Markdown of that page

Notice the malformed HTML:

This is what I have in the template:

    <h3>Content</h3>
    {{ .Content }}

    {{ if (not .Params.disable_debug) }}
        {{ with .Resources }}
            <hr />
            <h3>Resources</h3>

And this is the HTML around the part where the .Content ends:

<table>
<thead>
<tr>
<th>Inside <code>&lt;ORG_FILE_DIR&gt;</code></th>
<th>Copied-to location inside BUNDLE</th>
<th>Explanation</th>
</tr>
</thead>

<tbody>
<tr>
<td><code>&lt;ORG_FILE_DIR&gt;/bar/baz/foo.png</code></td>
<td><code>&lt;HUGO_BASE_DIR&gt;/content/&lt;SECTION&gt;/&lt;BUNDLE&gt;/bar/baz/foo.png</code></td>
<td>Even if the <strong>outside</strong> path does not have <code>/static/</code>, it is still inside the same dir as the Org file, 

    
        
            <hr />
            <h3>Resources</h3>

Notice that the ending portion of the table got truncated!

If I roll back to Hugo 0.40, this bug is gone. See that that table is generated in entirety:

<table>
<thead>
<tr>
<th>Inside <code>&lt;ORG_FILE_DIR&gt;</code></th>
<th>Copied-to location inside BUNDLE</th>
<th>Explanation</th>
</tr>
</thead>

<tbody>
<tr>
<td><code>&lt;ORG_FILE_DIR&gt;/bar/baz/foo.png</code></td>
<td><code>&lt;HUGO_BASE_DIR&gt;/content/&lt;SECTION&gt;/&lt;BUNDLE&gt;/bar/baz/foo.png</code></td>
<td>Even if the <strong>outside</strong> path does not have <code>/static/</code>, it is still inside the same dir as the Org file, so the directory structure is preserved.</td>
</tr>
</tbody>
</table>

<p>See <a href="https://ox-hugo.scripter.co/test/images-in-content/page-bundle-images-in-same-dir/">this other test</a> for an example.</p>


    
        
            <hr />
            <h3>Resources</h3>

How to recreate this issue

git clone --recursive -j8 https://github.com/kaushalmodi/ox-hugo
cd ox-hugo/test/site
hugo -b https://ox-hugo.scripter.co/test/
HTML referenced in this report: ./public/bundles/page-bundle-a/index.html.

I have yet been unable to create a smaller reproducible version of this bug. But it is 100% reproducible using the above steps.

The bug is absent in Hugo 0.40
The bug is present in Hugo 0.40.1 onwards.

The text was updated successfully, but these errors were encountered:

kaushalmodi · 2018-05-07T17:04:35Z

I wish I could understand the changes in this commit (288c396) that's the only functional difference between 0.40 and 0.40.1. But that's what introduced this issue.

bep · 2018-05-07T17:18:10Z

For the future, please keep bug titles in a neutral tone. Using movie style titles to draw attention really don't work.

kaushalmodi · 2018-05-07T17:21:57Z

I needed to make a point that this was serious; caused content loss. It's not a big deal for my test site, but could cause a problem with professional sites.

I am not sure what was "movie title like" in the old title.. I mentioned key points of information:

regression
since 0.40.1
serious, as causes content loss
bug description

bep · 2018-05-07T17:25:19Z

I cannot see in the example above that the .Content is chopped off? You say it is chopped off after the content?

kaushalmodi · 2018-05-07T17:32:54Z

ou say it is chopped off after the content?

No, the trailing portion of the .Content is chopped.

If you look at the Markdown source:

|----------------------------------|--------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------|
| `<ORG_FILE_DIR>/bar/baz/foo.png` | `<HUGO_BASE_DIR>/content/<SECTION>/<BUNDLE>/bar/baz/foo.png` | Even if the **outside** path does not have `/static/`, it is still inside the same dir as the Org file, so the directory structure is preserved. |

See [this other test](/images-in-content/page-bundle-images-in-same-dir/) for an example.

you will see that the HTML contains the content only up to "it is still inside the same dir as the Org file,":

<td>Even if the <strong>outside</strong> path does not have <code>/static/</code>, it is still inside the same dir as the Org file,

This part from the Markdown file (towards the end) is missing in the HTML:

so the directory structure is preserved. |

See [this other test](/images-in-content/page-bundle-images-in-same-dir/) for an example.

Note the "good HTML" snippet generated using Hugo 0.40 has this portion:

so the directory structure is preserved.</td>
</tr>
</tbody>
</table>

<p>See <a href="https://ox-hugo.scripter.co/test/images-in-content/page-bundle-images-in-same-dir/">this other test</a> for an example.</p>

which is absent in the live example.

As the table-closing is chopped off, the "Resources" heading which should be a separate h3 heading ends up in that incomplete table:

bep · 2018-05-07T17:33:08Z

Also, I don't understand how to build the above project (seems to need Emacs and vc-git etc).

kaushalmodi · 2018-05-07T17:34:32Z

Sorry, I did an edit of the steps few minutes back.. try:

git clone --recursive -j8 https://github.com/kaushalmodi/ox-hugo
cd ox-hugo/test/site
hugo -b https://ox-hugo.scripter.co/test/
HTML referenced in this report: ./public/bundles/page-bundle-a/index.html.

Screencast (asciinema)

bep · 2018-05-07T18:21:38Z

Yes, I can reproduce it on your site. I understand the what, but not the why. I will investigate, and issue a patch release tomorrow. Thanks for the report.

And re. the issue title, a movie title is of type "Scary movie in theatres on May 8th"

Issue titles should be short and tell what it is. The when and how serious etc. may belong in the body text, but it should be obvious from the what. I see too many drama titles of type "Hugo 0.x seriously blew up my site." A bug is super-serious to the issue reporter, I understand that, but ...

kaushalmodi · 2018-05-07T18:23:23Z

A bug is super-serious to the issue reporter, I understand that, but ...

Understood.

Yes, I can reproduce it on your site. I understand the what, but not the why. I will investigate, and issue a patch release tomorrow. Thanks for the report.

Thanks!

kaushalmodi · 2018-05-07T18:32:05Z

Looks like this issue is on a lot more pages than I thought; surprised why no one else spotted this.

Another example:

Markdown | HTML (bug not live any more.. site rebuilt using the fix in PR)

Note that all content after "It is necessary to set the Hugo site config variable" below is lost:

**It is necessary to set the Hugo site config variable
`pygmentsCodeFences` to `true` for syntax highlighting to work for
fenced code blocks.**

[^fn:1]: Even if the user has set the `HUGO_CODE_FENCE` value to `t` (via variable, keyword or subtree property), the Hugo `highlight` shortcode will be used automatically instead of code fences if either (i) the user has chosen to either show the line numbers, or (ii) has chosen to highlight lines of code (See the `ox-hugo` documentation on [Source Blocks](https://ox-hugo.scripter.co/doc/source-blocks)).

Here's the truncated .Content HTML:

<p><strong>It is necessary to set the Hugo site config variable
<

        

        

        <hr />
        <a id="debug"></a>

Something special about me using <hr /> immediately after {{ .Content }}? :)

Fixes gohugoio#4706

bep · 2018-05-07T19:48:58Z

surprised why no one else spotted this.

Your site does not represent the average Hugo site.

If you could take the PR above for a spin and tell me if that fixes this issue.

kaushalmodi · 2018-05-07T19:58:26Z

If you could take the PR above for a spin and tell me if that fixes this issue.

I confirm that to fix this issue (I have now rebuilt my site using the fix in the PR. The only way to see the bug is to now use Hugo v0.40.1, v0.40.2 and generate the site locally). Thanks!

Your site does not represent the average Hugo site.

OK. Not sure what would represent an average Hugo site. The only thing different I am doing in this site is print out all the debug info and implement search. Apart from that, it's pretty bare bones. I don't even use any shortcodes other than may be figure (apart from few test pages where I test shortcodes).

In any case, I'd like to understand how my site turned out to be a corner case (if so). Hopefully looking at the tests you add helps me understand that.

Also note that this issue showed up only on doing hugo, not hugo server. So might be a useful piece to this puzzle too.

bep · 2018-05-07T20:04:00Z

The only thing different I am doing in this site is print out all the debug info and implement search. Apart from that, it's pretty bare bones.

I will look at this more tomorrow. This is a bug, but I suspect that it relates to your not very bare bones page output definitions:

▶ find content -name "*.md" | xargs grep outputs
content/search.md:outputs = ["html", "json"]
content/posts/output-html-and-json.md:outputs = ["html", "json"]
content/posts/output-html-and-json.md:tags = ["outputs", "json"]
content/posts/output-html-and-json.md:template lookup hierarchy for the JSON outputs to be created.
content/posts/output-empty.md:title = "Setting empty outputs is fine"
content/posts/output-empty.md:tags = ["outputs", "empty"]
content/posts/output-empty.md:will not set the `outputs` variable in the front-matter at all. So
content/posts/keyword-collection.md:outputs = ["html", "json"]
content/posts/keyword-collection.md:-   [X] `#+hugo_outputs`

kaushalmodi · 2018-05-07T20:12:09Z

Aha, to separate out the matches with "outputs" in that:

Files with `outputs` front-matter

> find .  -name "*.md" | xargs grep -P 'outputs ='
./posts/keyword-collection.md:outputs = ["html", "json"]
./posts/output-html-and-json.md:outputs = ["html", "json"]
./search.md:outputs = ["html", "json"]

The first two are my tests for setting outputs. The last is what implements the test-site-wide search.

`outputs` as a tag

> find . -name "*.md" | xargs grep -P 'tags.*"outputs?"'
./posts/output-empty.md:tags = ["outputs", "empty"]
./posts/output-html-and-json.md:tags = ["outputs", "json"]

Problem to use outputs as a tag name?

I will look at this more tomorrow.

Thank you! I'll stay tuned! :)

bep · 2018-05-07T20:15:33Z

Problem to use outputs as a tag name?

No, that was just me being lazy with the grep.

There have been one report of a site with truncated `.Content` after the Hugo `0.41` release. This commit fixes this so that race should not be possible anymore. It also adds a stress test with focus on content rendering and multiple output formats. Fixes gohugoio#4706

anthonyfok · 2018-05-08T13:07:23Z

Hi @bep! Sorry for the interruption, but caught a possible typo in your commit log:

There have been one report of a site with truncated .Content after the Hugo 0.41 release.

I guess you meant to say 0.40 or 0.40.1 release there? Many thanks!

There have been one report of a site with truncated `.Content` after the Hugo `0.40.1` release. This commit fixes this so that race should not be possible anymore. It also adds a stress test with focus on content rendering and multiple output formats. Fixes gohugoio#4706

bep · 2018-05-08T13:58:51Z

Just a quick note. I will create a patch release with this tomorrow.

Also note that even if the fix was semi-obvious reading the code, I have not been able to reproduce this outside of the synthetic test site from @kaushalmodi -- I have tested the output before/after this patch for a set of sites, including the pretty big Kubernetes site with 1000+ long content pages in both English and Chinese, with exact same result.

kaushalmodi · 2018-05-08T14:14:35Z

I will create a patch release with this tomorrow.

Thanks!

I have not been able to reproduce this outside of the synthetic test site from @kaushalmodi

That definitely makes me feel special 😁. But was the issue caused just because of those outputs front-matter in those 3 pages? Apparently no one/very few people use that front-matter?

Note that my test site is testing every Hugo front-matter, for the sake of my ox-hugo package.

bep · 2018-05-08T14:32:59Z

@kaushalmodi I tried to pin-point the exact error case, but I got tired. The tests in this area should be greatly improved as a result of this, and that is great.

There have been one report of a site with truncated `.Content` after the Hugo `0.40.1` release. This commit fixes this so that race should not be possible anymore. It also adds a stress test with focus on content rendering and multiple output formats. Fixes gohugoio#4706

kaushalmodi · 2018-05-08T14:35:47Z

I tried to pin-point the exact error case, but I got tired.

No problem. Thank you for the prompt fix, and in addition adding more test cases!

There have been one report of a site with truncated `.Content` after the Hugo `0.40.1` release. This commit fixes this so that race should not be possible anymore. It also adds a stress test with focus on content rendering and multiple output formats. Fixes gohugoio#4706

There have been one report of a site with truncated `.Content` after the Hugo `0.40.1` release. This commit fixes this so that race should not be possible anymore. It also adds a stress test with focus on content rendering and multiple output formats. Fixes #4706

github-actions · 2022-03-04T02:01:45Z

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

bep changed the title ~~hugo 0.40.1 introduced serious regression: Chops off trailing .Content~~ Chops off trailing .Content May 7, 2018

bep added the NeedsInvestigation label May 7, 2018

bep added Bug and removed NeedsInvestigation labels May 7, 2018

bep changed the title ~~Chops off trailing .Content~~ Chops off trailing .Content in a multioutput setup May 7, 2018

bep added a commit to bep/hugo that referenced this issue May 7, 2018

hugolib: Fix possible .Content cutoff

c257526

Fixes gohugoio#4706

bep mentioned this issue May 7, 2018

hugolib: Fix possible .Content cutoff #4707

Merged

bep mentioned this issue May 8, 2018

Issue with .Render from list templates/multiple output formats #4708

Closed

bep changed the title ~~Chops off trailing .Content in a multioutput setup~~ Chops off trailing .Content in one multioutput setup May 8, 2018

bep added this to the v0.40.3 milestone May 8, 2018

bep closed this as completed in #4707 May 8, 2018

github-actions bot added the Outdated label Mar 4, 2022

github-actions bot locked as resolved and limited conversation to collaborators Mar 4, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Chops off trailing .Content in one multioutput setup #4706

Chops off trailing .Content in one multioutput setup #4706

kaushalmodi commented May 7, 2018 •

edited

kaushalmodi commented May 7, 2018

bep commented May 7, 2018

kaushalmodi commented May 7, 2018

bep commented May 7, 2018

kaushalmodi commented May 7, 2018 •

edited

bep commented May 7, 2018

kaushalmodi commented May 7, 2018 •

edited

bep commented May 7, 2018

kaushalmodi commented May 7, 2018

kaushalmodi commented May 7, 2018 •

edited

bep commented May 7, 2018

kaushalmodi commented May 7, 2018 •

edited

bep commented May 7, 2018 •

edited

kaushalmodi commented May 7, 2018

bep commented May 7, 2018

anthonyfok commented May 8, 2018

bep commented May 8, 2018

kaushalmodi commented May 8, 2018

bep commented May 8, 2018

kaushalmodi commented May 8, 2018

github-actions bot commented Mar 4, 2022

Chops off trailing .Content in one multioutput setup #4706

Chops off trailing .Content in one multioutput setup #4706

Comments

kaushalmodi commented May 7, 2018 • edited

Bug example

How to recreate this issue

kaushalmodi commented May 7, 2018

bep commented May 7, 2018

kaushalmodi commented May 7, 2018

bep commented May 7, 2018

kaushalmodi commented May 7, 2018 • edited

bep commented May 7, 2018

kaushalmodi commented May 7, 2018 • edited

bep commented May 7, 2018

kaushalmodi commented May 7, 2018

kaushalmodi commented May 7, 2018 • edited

bep commented May 7, 2018

kaushalmodi commented May 7, 2018 • edited

bep commented May 7, 2018 • edited

kaushalmodi commented May 7, 2018

Files with outputs front-matter

outputs as a tag

bep commented May 7, 2018

anthonyfok commented May 8, 2018

bep commented May 8, 2018

kaushalmodi commented May 8, 2018

bep commented May 8, 2018

kaushalmodi commented May 8, 2018

github-actions bot commented Mar 4, 2022

kaushalmodi commented May 7, 2018 •

edited

kaushalmodi commented May 7, 2018 •

edited

kaushalmodi commented May 7, 2018 •

edited

kaushalmodi commented May 7, 2018 •

edited

kaushalmodi commented May 7, 2018 •

edited

bep commented May 7, 2018 •

edited

Files with `outputs` front-matter

`outputs` as a tag