Import JSON data into pug

[Matthias-Hermsdorf]

**Pug Version:** 2.0.1
**Node Version:** 8.x

Hi there,

we are building a design system and styleguide with pug. When developing we have a express server generating the html. Some gulp tasks watch over all the files, generate the css, js and browsersyncs.

Often texts are coming from jsons and in the first iterations die jsons got required in the express part and given as locale variables to the pug.

But that meens you have a lot of ghost like global variables in every page. I don’t know the available names of the variables and the jsons are located far away from the rest of the component.

We changed the process. Now the only object we make available for the pug pages is require. just give {require: require} as a local and you have the might of everything in the template.

At the beginning of a mixin I readFile the json. I readFile them because require is cached for the server runtime and I modify the Files on runtime. The performance drawback is acceptable.

I can’t just include the json, because I need the json in the JS part to select and modify the strings.

My main discomfort is the long path to the json. At the moment the data fetching looks like:

mixin something(key)
  - 
    let path = require('path');
    let fs = require('fs');
    let dataJson = JSON.parse(fs.readFileSync(path.normalize(appRootPath.toString()+"/Private/src/components/path/to/compoent/data.json")));
  h1=dataJson[key]

It works, but it could be shorter, like:

mixin something(key)
  - 
    let dataJson = readJson(”data.json");
  h1=dataJson[key]

My main Problem is, that readJson doesn’t know the location of the calling mixin. Every call is relative to the project root.

Is there a way to get to know the current position? A variable, a function or an entry in this would be nice. And I can write the readJson myself.

[ForbesLindesay]

No, there's not currently any way of getting that information. A plugin would probably be the best way to do this. You could use a postParse plugin which would get a full AST of the pug file, and should have access to the pug filename. Then you can replace JavaScript code as needed. Unfortunately none of this is well documented. https://github.com/pugjs/pug-plugin-debug-line should give you a little bit of an idea of how to get started, but you'll have to do lots of console.log calls.

[Matthias-Hermsdorf]

Thanks for the advice. I will try.

[lunelson]

@Matthias-Hermsdorf you might want to take a look at the way I configure locals in this file; the project is a pug-as-middleware approach using browsersync itself as the server (which in turn uses connnect):

https://github.com/lunelson/penny/blob/master/lib/util/pug-locals.js#L55-L60

The important part is that this module exports a function which binds require and other functions to the locals object as their context, giving them access to this.filename

[Matthias-Hermsdorf]

@lunelson penny is interessting and I've learned something reading your code, for example delete require.cache[absFile]; was new for me. But you can access with relFile only the Path relative to the entry site, not relative to the mixin.

@ForbesLindesay I've digged through pug and now I have an plugin

    'use strict';
    
    const walk = require('pug-walk');
    const path = require('path');
    
    const keyWord = "currentPath"
    const keyPattern = new RegExp(keyWord,"g");
    
    module.exports.preCodeGen = function (ast, options) {
        ast = JSON.parse(JSON.stringify(ast));
    
        ast = walk(ast, function before(node, replace) {
            //console.log(node)
            if (node.type == "Code") {
    
                if (node.val.indexOf(keyWord > -1)) {
                    let replacement =  '\''+path.dirname(path.normalize(node.filename))+'\''
                    // because path.normalize reduces the count of \ to two, the escaping and the symbol
                    // but we need 4, because after the replacement 2 must remain for one in the calling mixin. ugly
                    replacement = replacement.replace(/\\/g, "\\\\");
    
                    node.val = node.val.replace(keyPattern,replacement);
                    replace(node);
                }
            }
        });
    
        return ast;
    };

and can write

let mixJson = require(currentPath+"/mix.json");

But then I spoke to a teammate. He is refactoring our styleguide he builds the pages on startup. And later on he observes file changes and uses the dependency tree to rebuild only the neccessary files. Looking on the code he said: How should I observe this?

We brainstormed to a more general solution and found out we need a new ast.type, thinking of

mixin example
  use example.json as exampleJson
  - exampleJson.keyInJson should be here
  h1="and here too: "+exampleJson.key

If we build this, would you merge the pullrequest?

[ForbesLindesay]

Some similar things have actually been requested a few times (#2804, #2772, #2604). What I'd like to do is add an import statement type. It would follow the node.js resolution procedure (probably using https://www.npmjs.com/package/resolve). It should ideally compile into a require call of some sort. Your example would look like:

mixin example
  import exampleJson from './example.json'
  - exampleJson.keyInJson should be here
  h1="and here too: "+exampleJson.key

This is not a small amount of work, but if you are willing to do that, I will definitely be happy to review/merge pull requests, and offer help where I can. The work breaks down as:

• add a lexer token for "import statement" - possibly just use "babylon" to parse the entire statement?
• add an "ImportStatement" node in the parser
• decide whether this just transpiles to a require call, or whether it inlines the imported code. - maybe an option to choose between them.

If it transpiles to a require call:

This makes it relatively easy to support all of JavaScript and JSON and anything else you might want to import in one go. It also ensures that imported code doesn't need to be loaded again for every template in the application. It may make this feature unusable for people who use the output of pug client side directly without any bundling step though. It also means that pug won't be able to track dependencies of imported files.

• resolve the absolute path in the pug-load step
• output the appropriate require/import call in pug-code-gen - N.B. we probably want to "hoist" it outside the generated template function.

The question here is, do we try to figure out the correct relative path (relative to the output file) or just use absolute paths for when people use compileClient / compileFileClient.

We'll also need to make sure some sort of require function is available at runtime. This could be pug_import as part of pug-runtime but may need to be somewhat of a special case.

If it inlines the imported code:

This is probably more straightforward for JSON files, but harder for everything else. We could support other things via some kind of plugin system though.

• resolve and read the file in pug-load, recursively loading dependencies of JS files (this bits tricky to say the least)
• add an extra step to "link" that imported data into the resulting template

[christophervoigt]

Hi, I'm one of @Matthias-Hermsdorf teammates ✌️ 😄

Our new approach to get data from JSON files is this:

let fs = self.require('fs');
let data = JSON.parse(fs.readFileSync('/src/path/to/component/data.json'));

As Matthias mentioned, we wrote a npm script which manually compiles our pages and uses the pug-dependency package to build an importMap along the way. This map is important for our watch/rebuild process. You basically make changes to a pug file and only the other files, which use this file, get rebuild. This saves a ton of time during development, compared to a gulp task.

Currently there is no possibility to integrate JSON files in the importMap. So it requires a manual change (hit spacebar, hit backspace and ctrl+s 😄 ) on the pug mixin that uses the JSON. So for me there would be some kind of workflow approvement, but thats not important at all.

@ForbesLindesay according to your approach of having some kind of import statement:

import data from some.json

IMO this comes a bit to close to the current ES6 imports, which might end up confusing a lot of people, if they try to import other javascript functionalities with it.

Another approach to this would be the extention of the include functionality, like

include ./data.json as data

mixin something(key)
    -
        let json = JSON.parse(data);

    h1= json[key]

Instead of inlining the include content, it would assign the content string to a scoped variable.
Wouldn't this be easier to accomplish?

[lunelson]

FWIW I would suggest a more neutral word than require or import, both of which are terms that tend to imply parsing the file as programmatic or dynamic content.

load or read might be more inline with the fact that you are talking about static data.

[ForbesLindesay]

IMO this comes a bit to close to the current ES6 imports

That's exactly the point. This should work like ES6 imports if you import a JavaScript file, and let you import JSON too (just like node.js + babel would let you do currently)

[christophervoigt]

Sorry, I assumed we wouldn't want to reimplement ES6 import. I mean, isn't this functionality comming to node anyway? (someday)

If / When this happens, wouldn't it then be possible to do something like this?

mixin something(key)
    -
        import data from './data.json';

    h1= data[key]

[Matthias-Hermsdorf]

ES6 Modules in Node will be experimental in 10.1, see https://nodejs.org/api/esm.html which is pretty soon.

I'm cautious with the name "import" which would imply for me that this is the JS import, but it isn't.
And with require in the locals I can read the files I want (yes, with a absolut path, but that could be shorted with the pug plugin).

The pug language addon to import things seemed neccesary for me for the dependency resolution. but now I think it would be better to extenc the pug-dependency plugin. I could parse the JS blocks there and search for imports and requires, without modifing the pug.

Doubling the import has a bad smell for me. There is less benefit, but many side-effects.

[ForbesLindesay]

@chlorophyllkid That won't work for a few of reasons:

1. imports need to be at the top level, so we would have to detect the imports and hoist them - this is much easier if we make an "import statement" a pug construct, rather than simply a special JavaScript statement
2. We don't typically output a file, we normally output a Function to eval.
3. You could use dynamic imports, but they're async (which pug isn't) and would still need to be in a file, not evaled as part of a new Function call.

For these reasons, I would like to replicate what babel does by compiling the import statement to a CommonJS style require call.

[christophervoigt]

I guess you're right, a pug construct would work much better.
My suggestion probably wouldn't work at all. 😄

But I'd still rather implement the solution that inlines the imported code. Mainly because it fits our usecase well enough and seems to be easier done.

[ForbesLindesay]

That's fine. I would suggest that for now, we only support: import foo from './foo.json' but make it extendible to allow custom handling of other file formats.

[Matthias-Hermsdorf]

I would close this issue. I could solve my problem with the pug plugin to resolve long file path and then readFile or require them in an code block. Thanks for the discussion and help.

[noraj]

Does someone found a way to properly import json files in pug?

ES6 import mapping from './mapping.json' is treadted as a <import> HTML tag since it's not implemented

- import mapping from './mapping.json' triggers an error as we are not in a module as pug is not an ESM

node 16.13.2

- import mapping from './mapping.json' assert { type: 'json' } on nodejs 18.7.0

All workaround mentioned here doesn't work either.

Can we re-open this issue?

The only solution I found with gulp is the following:

in gulpfile.mjs:

import gulp from 'gulp';
const { series, parallel, src, dest, task } = gulp;
import gulpPug from 'gulp-pug';
import gulpData from 'gulp-data';

// compile pug templates into HTML and pass data in argument
function pug_src() {
    return src('pug/*.pug')
        .pipe(gulpData(function() {
            return JSON.parse(fs.readFileSync('temp/data.json'));
        }))
        .pipe(gulpData(function() {
            return JSON.parse(fs.readFileSync('pug/mapping.json'));
        }))
        .pipe(gulpPug({
            pretty: true,
        }))
        .pipe(dest('build/'));
};

if the mapping.json file looks like that

{
  "mapping": {
    "Tools": [
      "A A",
      "B B",
      "C C"
    ]
}

Then I can do that in pug:

                each category in mapping.Tools
                    +category(category)

But I have to load each JSON file globally and there are referenced directly by the their root nodes so if there is the same node name in two json files there will be a conflict since I can't import them and give them an alias.