Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LESS processing issue, have to run nikola twice #2495

Closed
russel opened this issue Sep 7, 2016 · 26 comments
Closed

LESS processing issue, have to run nikola twice #2495

russel opened this issue Sep 7, 2016 · 26 comments
Assignees
Labels
Milestone

Comments

@russel
Copy link

russel commented Sep 7, 2016

I am finding that I have to run "nikola build" twice when using LESS processing and bundles. My hypothesis is that on the first execution the bundle is created without the output of the execution of lessc on the LESS source. On the second run the bundling picks up the previously created CSS from the LESS source. You have to do a "nikola clean" to really see this happening.

@ralsina ralsina added the bug label Sep 11, 2016
@ralsina ralsina added this to the v7.8.1 milestone Sep 11, 2016
@ralsina ralsina self-assigned this Sep 11, 2016
@ralsina
Copy link
Member

ralsina commented Sep 11, 2016

Ok, so, here's why this happens, then let's see if we can find a way to fix it.

The bundling builds from files in output/. That's needed because we don't really know what CSS you are getting, it may come from files/, or the theme, or whatever, so instead of trying to calculate that, just let doit's dependency wranglers do their thing. So output/bundle.css is built by this task and depends on output/thing.css which has been copied and depends on theme/foo/thing.css

LESS and similar plugins work from sources in the theme. So, you have one task that copies
cache/less/thing.less depending on theme/foo/less/thing.less, and another that builds output/thing.css depending on cache/less/thing.less.

So, as you can see, if the bundle depends correctly on output/thing.css there should be a full dependency chain all the way to theme/foo/less/thing.less and it should work.

Since it doesn't, clearly one of the things I said above is not true. All that's needed is to reproduce the bug and figure out which one :-)

@felixfontein
Copy link
Contributor

@russel: can you check whether my PR (getnikola/plugins#168) fixes this problem for you?

@russel
Copy link
Author

russel commented Sep 22, 2016

@felixfontein Is this now in a release on PyPI so that a pip upgrade will bring the change to me?

@Kwpolska
Copy link
Member

Nikola plugins are not part of the main release. They are part of the plugins repo. You would still need to manually download them from @felixfontein’s fork.

@russel
Copy link
Author

russel commented Sep 22, 2016

@Kwpolska I feel a bit of an idiot, of course it doesn't come form PyPI. It would obviously be better just to reinstall from the plugins repository if that were possible, but it sounds like I have to do something manually.

@Kwpolska
Copy link
Member

Just grab less/less.py and/or sass/sass.py from that repository and put it in plugins/less and plugins/sass respectively.

@russel
Copy link
Author

russel commented Sep 22, 2016

Wilco.

@felixfontein
Copy link
Contributor

It's now merged, so you can also take the file from the https://github.com/getnikola/plugins repository. (Direct link: https://raw.githubusercontent.com/getnikola/plugins/master/v7/less/less.py)

@Kwpolska Kwpolska modified the milestones: v7.8.1, v7.8.2 Oct 13, 2016
@felixfontein
Copy link
Contributor

Any news on this @russel?

@russel
Copy link
Author

russel commented Oct 15, 2016

I did a "nikola plugin --upgrade" and I assume new stuff got downloaded, it is claiming to be v0.1. The build_less appears to happen before the create_bundles. However the first pass is not stable, you still have to run a second pass. So having done "nikola build" and have everything build a second "nikola build" cause:

Scanning posts......done!
.  render_tags:output/categories/index.html
.  render_listings:output/listings/index.html
.  render_archive:output/2016/index.html
.  render_archive:output/archive.html
.  render_indexes:output/index.html
.  render_galleries:output/galleries
.  render_galleries:output/galleries/index.html
.  render_galleries:output/galleries/rss.xml
.  render_pages:output/posts/201607271615_russel_websiteemerges.html
.  render_pages:output/stories/code_of_conduct.html
.  render_pages:output/posts/201609261440_russel_accuconfdates.html
.  render_pages:output/posts/201609121650_russel_accuconfwebsiteinaflask.html
.  render_pages:output/posts/201607281130_russel_youtubechannel.html
.  create_bundles:output/assets/css/all.css
.  create_bundles:output/assets/css/all-nocdn.css
.  sitemap:output/sitemap.xml

with a third "nikola build" doing nothing.

@felixfontein
Copy link
Contributor

Can you run nikola info create_bundles:output/assets/css/all.css and nikola info create_bundles:output/assets/css/all-nocdn.css? And the same for the LESS tasks?

@felixfontein
Copy link
Contributor

In particular, take a look whether the files generated by your LESS tasks are in the file_deps of the bundle creation tasks.

@russel
Copy link
Author

russel commented Oct 15, 2016

I think this is the data you are after:

|> nikola info create_bundles:output/assets/css/all.css
Scanning posts......done!

name:'create_bundles:output/assets/css/all.css'

file_dep:{   'output/assets/css/code.css',
    'output/assets/css/colorbox.css',
    'output/assets/css/custom.css',
    'output/assets/css/rst.css',
    'output/assets/css/theme.css'}

task_dep:['copy_assets', 'copy_files']

targets:['output/assets/css/all.css']

|> nikola info create_bundles:output/assets/css/all-nocdn.css
Scanning posts......done!

name:'create_bundles:output/assets/css/all-nocdn.css'

file_dep:{   'output/assets/css/bootstrap.css',
    'output/assets/css/code.css',
    'output/assets/css/colorbox.css',
    'output/assets/css/custom.css',
    'output/assets/css/rst.css',
    'output/assets/css/theme.css'}

task_dep:['copy_assets', 'copy_files']

targets:['output/assets/css/all-nocdn.css']

|> nikola info build_less:output/assets/css/custom.css
Scanning posts......done!

name:'build_less:output/assets/css/custom.css'

file_dep:{'cache/less/custom.less'}

task_dep:['prepare_less_sources']

targets:['output/assets/css/custom.css']

@felixfontein
Copy link
Contributor

Yes, thanks. Hmmm, that's strange. The data looks like it is correct: build_less creates the target output/assets/css/custom.css, and create_bundles has that file as a file_dep. Which means that doit must execute the build_less tasks before create_bundles, and if that file is changed by build_less, doit should ensure that create_bundles is run.

So either the dependencies of create_bundles are sometimes different (and don't mention `output/assets/css/custom.css', which would cause that you have to run nikola again), in which case it is a nikola bug, otherwise it is a doit bug. Since doit usually works well I'd guess/assume it is a nikola bug.

Can you try to run nikola info create_bundles:output/assets/css/all.css and nikola info build_less:output/assets/css/custom.css several times in a row, and check whether the output changes?

You can also try modifying the SCSS file slightly so it generates a different CSS file (like add a dummy rule at the end; some whitespace changes might not be enough) and then run:

  • nikola build ache/less/custom.less
  • nikola info -s build_less:output/assets/css/custom.css
  • nikola build build_less:output/assets/css/custom.css
  • nikola info -s create_bundles:output/assets/css/all.css
    and see whether the last output mentions that all.css should be rebuilt. I don't know if that helps, but maybe it provides some helpful output.

@felixfontein
Copy link
Contributor

Also, could you describe your setup a little bit? Which theme do you use, and are you using any other .less files besides custom.less? And where did you put custom.less? Then I can try creating a minipage which (hopefully, otherwise we can't debug it) has the same behavior...

@russel
Copy link
Author

russel commented Oct 16, 2016

I think I followed the protocol, I think the last line is a bit weird?

|> nikola info create_bundles:output/assets/css/all.css
Scanning posts......done!

name:'create_bundles:output/assets/css/all.css'

file_dep:{   'output/assets/css/code.css',
    'output/assets/css/colorbox.css',
    'output/assets/css/rst.css',
    'output/assets/css/theme.css'}

task_dep:['copy_assets', 'copy_files']

targets:['output/assets/css/all.css']

|> nikola info create_bundles:output/assets/css/all.css
Scanning posts......done!

name:'create_bundles:output/assets/css/all.css'

file_dep:{   'output/assets/css/code.css',
    'output/assets/css/colorbox.css',
    'output/assets/css/rst.css',
    'output/assets/css/theme.css'}

task_dep:['copy_assets', 'copy_files']

targets:['output/assets/css/all.css']

|> nikola info create_bundles:output/assets/css/all.css
Scanning posts......done!

name:'create_bundles:output/assets/css/all.css'

file_dep:{   'output/assets/css/code.css',
    'output/assets/css/colorbox.css',
    'output/assets/css/rst.css',
    'output/assets/css/theme.css'}

task_dep:['copy_assets', 'copy_files']

targets:['output/assets/css/all.css']

|> nikola info build_less:output/assets/css/custom.css
Scanning posts......done!

name:'build_less:output/assets/css/custom.css'

file_dep:{'cache/less/custom.less'}

task_dep:['prepare_less_sources']

targets:['output/assets/css/custom.css']

|> nikola info build_less:output/assets/css/custom.css
Scanning posts......done!

name:'build_less:output/assets/css/custom.css'

file_dep:{'cache/less/custom.less'}

task_dep:['prepare_less_sources']

targets:['output/assets/css/custom.css']

|> nikola info build_less:output/assets/css/custom.css
Scanning posts......done!

name:'build_less:output/assets/css/custom.css'

file_dep:{'cache/less/custom.less'}

task_dep:['prepare_less_sources']

targets:['output/assets/css/custom.css']

|> nikola info build_less:output/assets/css/custom.css
Scanning posts......done!

name:'build_less:output/assets/css/custom.css'

file_dep:{'cache/less/custom.less'}

task_dep:['prepare_less_sources']

targets:['output/assets/css/custom.css']

|> nikola build cache/less/custom.less
Scanning posts......done!
.  prepare_less_sources:cache/less/custom.less

|> nikola info -s build_less:output/assets/css/custom.css
Scanning posts......done!

name:'build_less:output/assets/css/custom.css'

file_dep:{'cache/less/custom.less'}

task_dep:['prepare_less_sources']

targets:['output/assets/css/custom.css']

Task is not up-to-date:
 * The following uptodate objects evaluate to false:
    - Change with config: {"cache_folder": "cache", "themes": ["themes/accuconf", "/home/users/russel/.local/lib/python3.5/site-packages/nikola/data/themes/bootstrap3-jinja", "/home/users/russel/.local/lib/python3.5/site-packages/nikola/data/themes/base-jinja", "/home/users/russel/.local/lib/python3.5/site-packages/nikola/data/themes/base"]} (args=[], kwargs={})
 * The following file dependencies have changed:
    - cache/less/custom.less
 * The following targets do not exist:
    - output/assets/css/custom.css

|> nikola info -s build_less:output/assets/css/custom.css
Scanning posts......done!

name:'build_less:output/assets/css/custom.css'

file_dep:{'cache/less/custom.less'}

task_dep:['prepare_less_sources']

targets:['output/assets/css/custom.css']

Task is not up-to-date:
 * The following uptodate objects evaluate to false:
    - Change with config: {"cache_folder": "cache", "themes": ["themes/accuconf", "/home/users/russel/.local/lib/python3.5/site-packages/nikola/data/themes/bootstrap3-jinja", "/home/users/russel/.local/lib/python3.5/site-packages/nikola/data/themes/base-jinja", "/home/users/russel/.local/lib/python3.5/site-packages/nikola/data/themes/base"]} (args=[], kwargs={})
 * The following targets do not exist:
    - output/assets/css/custom.css
 * The following file dependencies have changed:
    - cache/less/custom.less

|> nikola build build_less:output/assets/css/custom.css
Scanning posts......done!
.  prepare_less_sources:cache/less/targets
.  build_less:output/assets/css/custom.css

|> nikola info -s build_less:output/assets/css/custom.css
Scanning posts......done!

name:'build_less:output/assets/css/custom.css'

file_dep:{'cache/less/custom.less'}

task_dep:['prepare_less_sources']

targets:['output/assets/css/custom.css']

Task is up-to-date.

|> nikola info -s create_bundles:output/assets/css/all.css
Scanning posts......done!

name:'create_bundles:output/assets/css/all.css'

file_dep:{   'output/assets/css/code.css',
    'output/assets/css/colorbox.css',
    'output/assets/css/custom.css',
    'output/assets/css/rst.css',
    'output/assets/css/theme.css'}

task_dep:['copy_assets', 'copy_files']

targets:['output/assets/css/all.css']

Task is not up-to-date:
 * The following uptodate objects evaluate to false:
    - Change with config: {"1": {"cache_folder": "cache", "code_color_scheme": "default", "files_folders": {"files": ""}, "filters": {".php": ["<function php_template_injection at "]}, "output_folder": "output", "theme_bundles": {"assets/css/all-nocdn.css": ["bootstrap.css", "rst.css", "code.css", "colorbox.css", "theme.css", "custom.css"], "assets/css/all.css": ["rst.css", "code.css", "colorbox.css", "theme.css", "custom.css"], "assets/js/all-nocdn.js": ["jquery.min.js", "bootstrap.min.js", "jquery.colorbox-min.js", "moment-with-locales.min.js", "fancydates.js", "underscore-min.js"], "assets/js/all.js": ["jquery.colorbox-min.js", "moment-with-locales.min.js", "fancydates.js", "underscore-min.js"]}, "themes": ["themes/accuconf", "/home/users/russel/.local/lib/python3.5/site-packages/nikola/data/themes/bootstrap3-jinja", "/home/users/russel/.local/lib/python3.5/site-packages/nikola/data/themes/base-jinja", "/home/users/russel/.local/lib/python3.5/site-packages/nikola/data/themes/base"]}, "2": ["output/assets/css/rst.css", "output/assets/css/code.css", "output/assets/css/colorbox.css", "output/assets/css/theme.css", "output/assets/css/custom.css"]} (args=[], kwargs={})
 * The following file dependencies have changed:
    - output/assets/css/custom.css
 * The following file dependencies are missing:
    - output/assets/css/theme.css
    - output/assets/css/code.css
    - output/assets/css/colorbox.css
    - output/assets/css/rst.css
 * The following targets do not exist:
    - output/assets/css/all.css

@russel
Copy link
Author

russel commented Oct 16, 2016

The site is a part of the ACCU website, though I get similar problems with my personal site. The ACCU website code is public, the code for my site is not. The ACCU website is part Flask application part Nikola generated static pages; the repository hold two directories one for each bit.

https://github.com/ACCUConf/ACCUConf_Website

@felixfontein
Copy link
Contributor

I tried the ACCUConf website; every time I change themes/accuconf/less/custom.less, running nikola build compiles it and creates the bundles in one go. (I'm using lessc instead of lesscpy, though. Maybe lesscpy does some funny stuff which results in doit ignoring its changed output?)

@felixfontein
Copy link
Contributor

There's something else strange happening, though: when starting with the empty site (without cache and output folders), I have to run nikola build two times until everything's done. This happens for the following tasks:

.  render_tags:output/categories/index.html
.  render_archive:output/2016/index.html
.  render_archive:output/archive.html
.  render_indexes:output/index.html
.  render_pages:output/posts/201609261440_russel_accuconfdates.html
.  render_pages:output/posts/201609121650_russel_accuconfwebsiteinaflask.html
.  render_pages:output/posts/201607281130_russel_youtubechannel.html
.  render_pages:output/posts/201607271615_russel_websiteemerges.html
.  create_bundles:output/assets/css/all.css
.  create_bundles:output/assets/css/all-nocdn.css

For all but the last two, a changed uptodate dependency causes this. Why it changes, though, I don't know.

For the last two, this happens because the dependency output/assets/css/custom.css was added. This is due to the way Nikola's LESS plugin collects dependencies.

@felixfontein
Copy link
Contributor

felixfontein commented Oct 16, 2016

Ah. The change for the first tasks is that has_custom_css changes from False in the first pass to True in the second pass, which is due to output/assets/css/custom.css not yet existing when nikola build is ran the first time. That's because Nikola doesn't know who (or if anyone at all) will create that file when it needs that information first; so this is hard to solve without two passes.

@felixfontein
Copy link
Contributor

The best / only way to solve this is probably to extend the Task pluing interface by a method which allows to query whether the plugin generates a certain .css file in the output. (That would also remove some other hacks, such as the code.css detection.) It might force task plugins to do some more work to find out whether they create that file, though, especially because this has to happen before calling gen_tasks. I'm not sure whether it is a good idea to implement something like that.

@ralsina, @Kwpolska: what do you think? Or do you have other ideas on how to fix this?

@russel
Copy link
Author

russel commented Oct 16, 2016

The alternative is to say that, like XeLaTeX, one run is never enough, always run things twice.

Seriously, if the fix is tortuously complicated, it is fine to say that with some plugins you have to run the build twice to get over phasing issues.

@Kwpolska Kwpolska modified the milestones: v7.8.3, v7.8.2 Jan 8, 2017
@Kwpolska Kwpolska modified the milestones: v7.8.4, v7.8.5 Mar 26, 2017
@ralsina
Copy link
Member

ralsina commented May 3, 2017

Hmmmm... we can use plugin dependency injection. If a plugin generates CSS, it could install itself as a dependency for less, or something like that. Or the other way around, I have not read this very thoroughly.

@Kwpolska Kwpolska modified the milestones: v7.8.5, v7.8.6 May 14, 2017
@Kwpolska Kwpolska modified the milestones: v7.8.6, v7.8.7 May 26, 2017
@Kwpolska Kwpolska modified the milestones: v7.8.7, v7.8.8 Jun 4, 2017
@Kwpolska Kwpolska modified the milestones: v7.8.8, v7.8.9, v8.0.0 Jun 12, 2017
@Kwpolska
Copy link
Member

Kwpolska commented Sep 5, 2018

This is not going to be easy/possible to fix, running nikola build twice is the best solution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants