Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Allow files to be preprocessed before inclusion. #2

Closed
wants to merge 2 commits into from

2 participants

@timoxley

This builds upon the functionality in #1, adding the ability to run a file's content through a preprocessor before being added to the JSON. My use case I'm building a backend-less website, where the data is simply defined in json, and I'd like to be able to easily include the content of files in non-JSON formats (e.g. markdown) in my json file.

If there's anything you'd like me to do to make these PRs better, let me know.

@tj
Owner
tj commented

this is cool, i like the intent but I think that maybe we could use regular plugins per file type, one for markdown etc:

{ "readme": "markdown readme.md" }

or hell maybe even have an option:

.use(markdown({ prefix: false }))

{ "readme": "readme.md" }
@timoxley

Fair call. That's a cleaner api. Just detect the filetype and load any plugin registered for it.

@tj
Owner
tj commented

yeah screw the prefix, if they opt-in with .use(eson.markdown) then just check the extension, sounds reasonable to me. Im not sure how well npm's optional dependences work, maybe we should use that but there are so many markdown libs out there :s

@timoxley

Perhaps using a more succinct include syntax would make it tidier without increasing risk of accidental inclusion. e.g.

{ "readme": "<< readme.md" }

Also, the reason the processing functions are attached to the include plugin is because they don't really make sense without it. Perhaps a better syntax would be:

eson.include.extensions.md  = markdown()

eson()
.use(eson.include)
.use(eson.include.markdown)

Or simpler:

eson()
.use(eson.include)
.use(eson.markdown)

Or even better:

eson()
.use(eson.markdown)

In this final case the include plugin is implicit but it makes sense that only .md files can be included.

Most common use case will probably be .use(eson.json) anyway.

@tj
Owner
tj commented

why do they need include? it's just a tiny bit of logic, we can duplicate that stuff

@timoxley

Sure, but you'd lose the ability to use globbing to create arrays of data, or maps of files. See #1.

@tj
Owner
tj commented

meh that's alright, I dont have any reason to do globbing personally, not sure if others are really using this lib haha would be nice to have some more use-cases

@timoxley

I'm trying to use this to create a static single-page-app site generator, with no db component at all. just create a hierarchy of json files as your 'database', and eson pulls the db together as a single json file. The json/templates/js/css is shipped off to S3 and voila, single-page web app which uses something like http://angryamoeba.co.uk/spahql-announce/ for clientside 'db'. Basically, eson serves as the tool to build a 'database' up from a folder hierarchy of assets. This is mainly so I don't have to build any UI for the client to create data in the system, they just create json files and put them in a folder. And if I do want to add a UI, all the UI has to do is generate json files.

e.g. Create a product catalogue by simply dumping product information as json files in a products folder:

catalogue.json:

{
  "products": "include [products/*]"
}

products/conditioner.json:

   {
      "sku": "1",
      "name": "Conditioner"
   }

products/shampoo.json:

   {
      "sku": "2",
      "name": "Shampoo",
      "description": "include shampoo.md"
   }

products/shampoo.md:

#### Good shampoo, cleans hair

catalogue.json after an eson build:

{
  "products": [
    { "sku": "1",
      "name": "Conditioner"},
    { "sku": "2",
      "name": "Shampoo",
      "description": "<h4>Good shampoo, cleans hair</h4>"}
  ]
}

Seems like a good idea to me.

@tj
Owner
tj commented

ahhh I see that makes things a bit clearer for me, very cool, i'd definitely like to avoid a markdown dep ideally, so many out there and some are better than others depending on what you're really doing

@timoxley

Yep, including anything other than json files should be an external module e.g. eson-markdown, and its just listed in the readme.

@timoxley

um, bump

@timoxley

actually, too many conflicts to fix to bring this up to date. Closing.

@timoxley timoxley closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
This page is out of date. Refresh to see the latest.
View
109 Readme.md
@@ -90,7 +90,7 @@ yields:
### eson.include
- The include plugin allows you to literally include other JSON files. This works in
+ The include plugin allows you to literally include other JSON files. This works in
both arrays and object literals, and loads relative to the callee's file. For example:
```js
@@ -105,6 +105,111 @@ yields:
{ prod: { whatever: 'is', within: 'config/production.json' }}
```
+ You can also include multiple files via a glob. This has a special syntax and works in one of three ways.
+
+Consider a config folder containing the following two files:
+
+*database.json:*
+```json
+{"db", "redis"}
+```
+*app.json:*
+```json
+{"listen", 3000}
+```
+
+##### Merging multiple files into one:
+
+```js
+
+eson()
+ .use(eson.include)
+ .parse('{ "prod": "include config/*" }');
+```
+yields:
+
+```js
+{
+ prod: {
+ db: "redis",
+ listen: 3000
+ }
+}
+```
+
+##### Collect files into a map, keyed by filename:
+
+```js
+
+// use curly brackets to collect as a map
+eson()
+ .use(eson.include)
+ .parse('{ "prod": "include { config/* }" }');
+
+```
+
+yields:
+
+
+```js
+{
+ prod: {
+ database: {
+ db: "redis"
+ },
+ app: {
+ listen: 3000
+ }
+ }
+}
+
+```
+
+##### Collect files as an array:
+
+
+```js
+
+// use square brackets to collect as an array
+eson()
+ .use(eson.include)
+ .parse('{ "prod": "include [ config/* ]" }');
+
+```
+
+yields:
+
+```js
+{
+ prod: [
+ {db: "redis"},
+ {listen: 3000}
+ ]
+}
+```
+#### Preprocessing
+
+You can preprocess files based on file extension before including them in the JSON, just remember you must return a valid JSON value.
+```js
+// register a function to handle .md file content
+include.extensions.md = function(fileContent) {
+ var markdown = require('markdown')
+ // JSON stringify ensures content is a valid JSON value
+ return JSON.stringify(markdown.markdown.toHTML(fileContent))
+}
+
+eson()
+ .use(eson.include)
+ .parse('{ "readme": "include readme.md" }');
+
+```
+
+yields:
+
+```js
+{ readme: "<h1>Markdown</h1><p>Content</p>" }
+```
+
### eson.replace(str, val)
The replace plugin allows you to replace arbitrary substrings, useful
@@ -178,4 +283,4 @@ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
-SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
View
88 lib/plugins/include.js
@@ -4,16 +4,92 @@
var path = require('path')
, dirname = path.dirname
- , join = path.join;
-
+ , join = path.join
+ , glob = require('glob');
/**
* Include another JSON file,
* for example "include config/permissions".
*/
-module.exports = function(key, val, parser){
- var m = /^include +(.+)$/.exec(val);
+exports = module.exports = function(key, val, parser){
+ var m = /^include +([\{\[]?) *(.*?) *([\}\]]?)$/.exec(val);
+ var simple = /^include +([^\{\[].*)$/.exec(val)
+ var brace, relativePath
+ if (simple) m = simple;
if (!m) return;
- var path = join(dirname(parser.path), m[1] + '.json');
- return parser.clone().read(path);
+ if (!simple) {
+ if ((m[1] == "" && m[3] != "") ||
+ (m[1] != "" && m[1].charCodeAt(0) + 2 !== m[3].charCodeAt(0))) {
+ return; // braces don't match
+ }
+ brace = m[1];
+ relativePath = m[2];
+ } else {
+ brace = "";
+ relativePath = m[1];
+ }
+ var result;
+ // if not globbing, can leave off .json
+ if (path.extname(relativePath) === '') {
+ relativePath += '.json'
+ }
+ var paths = glob.sync(join(dirname(parser.path), relativePath));
+ paths = paths.sort();
+ if (!paths.length) {
+ console.warn('Warning: No paths match:', val);
+ }
+ switch (brace) {
+ case "":
+ if (!paths.length) return ""
+ return paths.map(function(absolutePath) {
+ return parser.clone().parse(exports.getData(absolutePath));
+ }).reduce(function(previous, data) {
+ for (var key in data) {
+ previous[key] = data[key];
+ }
+ return previous;
+ })
+ break;
+ case "{":
+ if (!paths.length) return {}
+ return paths.map(function(absolutePath) {
+ // get file name without extension
+ var key = path.basename(absolutePath);
+ key = key.slice(0, key.indexOf(path.extname(absolutePath)));
+
+ var result = {};
+ result[key] = parser.clone().parse(exports.getData(absolutePath));
+ return result;
+ }).reduce(function(previous, data) {
+ for (var key in data) {
+ previous[key] = data[key];
+ }
+ return previous;
+ })
+ break;
+ case '[':
+ if (!paths.length) return []
+ return paths.map(function(absolutePath) {
+ return parser.clone().parse(exports.getData(absolutePath));
+ })
+ break;
+ default:
+ // shouldn't get here
+ console.warn('No Match:', brace);
+ }
+}
+
+exports.extensions = {
+ json: function(data) {
+ return data;
+ }
}
+
+exports.getData = function(relativePath) {
+ var extname = path.extname(relativePath);
+ var fileContents = require('fs').readFileSync(relativePath, 'utf8')
+ var processFunction = exports.extensions[extname.slice(1)] || function() { return '""'; }
+ return processFunction(fileContents);
+}
+
+
View
1  test/fixtures/config/database.json
@@ -0,0 +1 @@
+{"db": "redis"}
View
1  test/fixtures/config/http.json
@@ -0,0 +1 @@
+{"listen": 8000}
View
8 test/fixtures/include_glob.json
@@ -0,0 +1,8 @@
+{
+ "app-config": "include config/{database,http}",
+ "user-config": "include { config/{permissions,roles}}",
+ "users": "include [ users/* ]",
+ "invalid-glob": "include doesnotexists*",
+ "invalid-map": "include {doesnotexists*}",
+ "invalid-array": "include [doesnotexists*]"
+}
View
3  test/fixtures/include_preprocessor.json
@@ -0,0 +1,3 @@
+{
+ "readme": "include readme.md"
+}
View
3  test/fixtures/readme.md
@@ -0,0 +1,3 @@
+# Read me
+
+OMG
View
1  test/fixtures/users/dave.json
@@ -0,0 +1 @@
+{"username": "Dave"}
View
1  test/fixtures/users/tobi.json
@@ -0,0 +1 @@
+{"username": "Tobi"}
View
27 test/include.js
@@ -16,4 +16,29 @@ describe('include', function(){
["admin", "guest"]
]);
})
-})
+ it('should parse files matching glob', function() {
+ Parser()
+ .use(include)
+ .read('test/fixtures/include_glob.json')
+ .should.eql({
+ "app-config": { "db": "redis", "listen": 8000 },
+ "user-config": { "permissions": { "view videos": "guest", "delete videos": "admin" }, "roles": ["admin", "guest"] },
+ "users": [ {"username": "Dave"}, {"username": "Tobi"} ],
+ "invalid-glob": "",
+ "invalid-map": {},
+ "invalid-array": []
+ })
+ })
+ it('should parse files using preprocesor', function() {
+ include.extensions.md = function(fileContent) {
+ // pretend this is a markdown parser
+ return JSON.stringify(fileContent.toUpperCase())
+ }
+ Parser()
+ .use(include)
+ .read('test/fixtures/include_preprocessor.json')
+ .should.eql({
+ "readme": "# READ ME\n\nOMG\n"
+ })
+ })
+})
Something went wrong with that request. Please try again.