Raw sassdoc data is not valid JSON #181

Closed
alademann opened this Issue Aug 21, 2014 · 29 comments

Comments

Projects
None yet
4 participants
@alademann
Contributor

alademann commented Aug 21, 2014

@HugoGiraudel According to this section in the readme, the raw data is made available.

However - I am unable to easily create a JSON file out of this, because the output is not valid JSON. Some key values are single-quoted, some have no quotes at all.

For instance - nothing happens when I try this:

sassdoc.parse('./sass').then(function (items) {
    console.log(JSON.stringify(items));
});

And the raw data itself - looks like this: (Which even the syntax highlighter github is using recognizes as invalid JSON)

{ 
    variable: { 
        'breakpoint-no-query-fallbacks': { 
            description: '',
            context: [Object],
            type: [Object],
            since: [Object],
            requires: [Object],
            link: [Object],
            access: [Object],
            group: [Object],
            file: [Object] 
        },
        'font-size-xs': { 
            description: '',
            context: [Object],
            type: [Object],
            since: [Object],
            requires: [Object],
            access: [Object],
            group: [Object],
            file: [Object] 
        },
        modules: { 
            description: '',
            context: [Object],
            ignore: [],
            access: [Object],
            group: [Object],
            file: [Object] 
        } 
    },
    mixin: { 
        'triangle-direction': { 
            description: '...',
            context: [Object],
            since: [Object],
            requires: [Object],
            parameters: [Object],
            throws: [Object],
            access: [Object],
            group: [Object],
            file: [Object],
            usedBy: [Object] 
        },
        triangles: { 
            description: '...',
            context: [Object],
            since: [Object],
            requires: [Object],
            parameters: [Object],
            access: [Object],
            group: [Object],
            file: [Object] 
        },
        'btn-reset': { 
            description: '...',
            context: [Object],
            since: [Object],
            requires: [Object],
            access: [Object],
            group: [Object],
            file: [Object],
            usedBy: [Object] 
        },
        'hide-text': { 
            description: '...',
            context: [Object],
            since: [Object],
            access: [Object],
            group: [Object],
            file: [Object] 
        } 
    },
    function: { 
        color: { 
            description: '...',
            context: [Object],
            access: [Object],
            since: [Object],
            requires: [Object],
            parameters: [Object],
            returns: [Object],
            group: [Object],
            file: [Object],
            usedBy: [Object] 
        },
        'state-variation-text': { 
            description: '...',
            context: [Object],
            since: [Object],
            requires: [Object],
            parameters: [Object],
            throws: [Object],
            returns: [Object],
            access: [Object],
            group: [Object],
            file: [Object] 
        } 
    } 
}

Note that keys containing - are quoted (using single quotes) - and those that do not have dashes, have no quotes. (It all should be double-quoted for valid JSON). I'm not sure if there are other reasons why the JSON is invalid, possibly quotes within the descriptions / other fields or something... but at a minimum, I would expect the raw data to return the ACTUAL objects within objects - not [Object].

As a consumer, I shouldn't have to write my own parser for the parser :)

@HugoGiraudel

This comment has been minimized.

Show comment
Hide comment
@HugoGiraudel

HugoGiraudel Aug 21, 2014

Member

As a consumer, I shouldn't have to write my own parser for the parser :)

Agreed, it's obviously a bug. We'll see what we can do. :)

Member

HugoGiraudel commented Aug 21, 2014

As a consumer, I shouldn't have to write my own parser for the parser :)

Agreed, it's obviously a bug. We'll see what we can do. :)

@HugoGiraudel HugoGiraudel added the Bug label Aug 21, 2014

@alademann

This comment has been minimized.

Show comment
Hide comment
@alademann

alademann Aug 21, 2014

Contributor

For the sake of clarity - here is that same example data, in a valid JSON format (excluding the contents of each [Object] - which should still be there, obviously.

{ 
    "variable": { 
        "breakpoint-no-query-fallbacks": { 
            "description": "...",
            "context": {},
            "type": {},
            "since": {},
            "requires": {},
            "link": {},
            "access": {},
            "group": {},
            "file": {} 
        },
        "font-size-xs": { 
            "description": "...",
            "context": {},
            "type": {},
            "since": {},
            "requires": {},
            "access": {},
            "group": {},
            "file": {} 
        },
        "modules": { 
            "description": "...",
            "context": {},
            "ignore": [],
            "access": {},
            "group": {},
            "file": {} 
        } 
    },
    "mixin": { 
        "triangle-direction": { 
            "description": "...",
            "context": {},
            "since": {},
            "requires": {},
            "parameters": {},
            "throws": {},
            "access": {},
            "group": {},
            "file": {},
            "usedBy": {} 
        },
        "triangles": { 
            "description": "...",
            "context": {},
            "since": {},
            "requires": {},
            "parameters": {},
            "access": {},
            "group": {},
            "file": {} 
        },
        "btn-reset": { 
            "description": "...",
            "context": {},
            "since": {},
            "requires": {},
            "access": {},
            "group": {},
            "file": {},
            "usedBy": {} 
        },
        "hide-text": { 
            "description": "...",
            "context": {},
            "since": {},
            "access": {},
            "group": {},
            "file": {} 
        } 
    },
    "function": { 
        "color": { 
            "description": "...",
            "context": {},
            "access": {},
            "since": {},
            "requires": {},
            "parameters": {},
            "returns": {},
            "group": {},
            "file": {},
            "usedBy": {} 
        },
        "state-variation-text": { 
            "description": "...",
            "context": {},
            "since": {},
            "requires": {},
            "parameters": {},
            "throws": {},
            "returns": {},
            "access": {},
            "group": {},
            "file": {} 
        } 
    } 
}
Contributor

alademann commented Aug 21, 2014

For the sake of clarity - here is that same example data, in a valid JSON format (excluding the contents of each [Object] - which should still be there, obviously.

{ 
    "variable": { 
        "breakpoint-no-query-fallbacks": { 
            "description": "...",
            "context": {},
            "type": {},
            "since": {},
            "requires": {},
            "link": {},
            "access": {},
            "group": {},
            "file": {} 
        },
        "font-size-xs": { 
            "description": "...",
            "context": {},
            "type": {},
            "since": {},
            "requires": {},
            "access": {},
            "group": {},
            "file": {} 
        },
        "modules": { 
            "description": "...",
            "context": {},
            "ignore": [],
            "access": {},
            "group": {},
            "file": {} 
        } 
    },
    "mixin": { 
        "triangle-direction": { 
            "description": "...",
            "context": {},
            "since": {},
            "requires": {},
            "parameters": {},
            "throws": {},
            "access": {},
            "group": {},
            "file": {},
            "usedBy": {} 
        },
        "triangles": { 
            "description": "...",
            "context": {},
            "since": {},
            "requires": {},
            "parameters": {},
            "access": {},
            "group": {},
            "file": {} 
        },
        "btn-reset": { 
            "description": "...",
            "context": {},
            "since": {},
            "requires": {},
            "access": {},
            "group": {},
            "file": {},
            "usedBy": {} 
        },
        "hide-text": { 
            "description": "...",
            "context": {},
            "since": {},
            "access": {},
            "group": {},
            "file": {} 
        } 
    },
    "function": { 
        "color": { 
            "description": "...",
            "context": {},
            "access": {},
            "since": {},
            "requires": {},
            "parameters": {},
            "returns": {},
            "group": {},
            "file": {},
            "usedBy": {} 
        },
        "state-variation-text": { 
            "description": "...",
            "context": {},
            "since": {},
            "requires": {},
            "parameters": {},
            "throws": {},
            "returns": {},
            "access": {},
            "group": {},
            "file": {} 
        } 
    } 
}
@alademann

This comment has been minimized.

Show comment
Hide comment
@alademann

alademann Aug 21, 2014

Contributor

Thanks for taking a look @HugoGiraudel !

Contributor

alademann commented Aug 21, 2014

Thanks for taking a look @HugoGiraudel !

@valeriangalliat

This comment has been minimized.

Show comment
Hide comment
@valeriangalliat

valeriangalliat Aug 21, 2014

Member

Are you sure you didn't execute console.log(items)?

Because I never saw JSON.stringify output single quoted (or not even quoted) keys, and the [object Object] strings is typically what console.log outputs at a limit nesting level when you log an object.

Member

valeriangalliat commented Aug 21, 2014

Are you sure you didn't execute console.log(items)?

Because I never saw JSON.stringify output single quoted (or not even quoted) keys, and the [object Object] strings is typically what console.log outputs at a limit nesting level when you log an object.

@alademann

This comment has been minimized.

Show comment
Hide comment
@alademann

alademann Aug 21, 2014

Contributor

That output is for console.log(items) @valeriangalliat. I get no output of any kind when executing console.log(JSON.stringify(items)).

Contributor

alademann commented Aug 21, 2014

That output is for console.log(items) @valeriangalliat. I get no output of any kind when executing console.log(JSON.stringify(items)).

@valeriangalliat

This comment has been minimized.

Show comment
Hide comment
@valeriangalliat

valeriangalliat Aug 21, 2014

Member

Oh okay. You don't even get an exception?

Member

valeriangalliat commented Aug 21, 2014

Oh okay. You don't even get an exception?

@alademann

This comment has been minimized.

Show comment
Hide comment
@alademann

alademann Aug 21, 2014

Contributor

nope

Contributor

alademann commented Aug 21, 2014

nope

@alademann

This comment has been minimized.

Show comment
Hide comment
@alademann

alademann Aug 21, 2014

Contributor

Also - if console.log(items) doesn't give you access to the raw data - then the README in this repo shouldn't instruct people to use it.

Contributor

alademann commented Aug 21, 2014

Also - if console.log(items) doesn't give you access to the raw data - then the README in this repo shouldn't instruct people to use it.

@HugoGiraudel

This comment has been minimized.

Show comment
Hide comment
@HugoGiraudel

HugoGiraudel Aug 21, 2014

Member

Also - if console.log(items) doesn't give you access to the raw data - then the README in this repo shouldn't instruct people to use it.

It does.

Member

HugoGiraudel commented Aug 21, 2014

Also - if console.log(items) doesn't give you access to the raw data - then the README in this repo shouldn't instruct people to use it.

It does.

@valeriangalliat

This comment has been minimized.

Show comment
Hide comment
@valeriangalliat

valeriangalliat Aug 21, 2014

Member

The function don't return undefined nor null? It means JSON.stringify is still running then. It means the data contains circular references.

Member

valeriangalliat commented Aug 21, 2014

The function don't return undefined nor null? It means JSON.stringify is still running then. It means the data contains circular references.

@alademann

This comment has been minimized.

Show comment
Hide comment
@alademann

alademann Aug 21, 2014

Contributor

@HugoGiraudel not if it cannot access the nested objects as @valeriangalliat said. It doesn't do me any good if I can't have all the data.

Contributor

alademann commented Aug 21, 2014

@HugoGiraudel not if it cannot access the nested objects as @valeriangalliat said. It doesn't do me any good if I can't have all the data.

@HugoGiraudel

This comment has been minimized.

Show comment
Hide comment
@HugoGiraudel

HugoGiraudel Aug 21, 2014

Member

Also - if console.log(items) doesn't give you access to the raw data - then the README in this repo shouldn't instruct people to use it.

The data does contain some circular references.

Member

HugoGiraudel commented Aug 21, 2014

Also - if console.log(items) doesn't give you access to the raw data - then the README in this repo shouldn't instruct people to use it.

The data does contain some circular references.

@valeriangalliat

This comment has been minimized.

Show comment
Hide comment
@valeriangalliat

valeriangalliat Aug 21, 2014

Member

@alademann You can have all the data but you can't serialize it for now.

Member

valeriangalliat commented Aug 21, 2014

@alademann You can have all the data but you can't serialize it for now.

@HugoGiraudel

This comment has been minimized.

Show comment
Hide comment
@HugoGiraudel

HugoGiraudel Aug 21, 2014

Member

@HugoGiraudel not if it cannot access the nested objects as @valeriangalliat said. It doesn't do me any good if I can't have all the data.

You do have all the data. You just can't serialize it because of circular dependencies. That doesn't prevent you from doing things with the data. Hence the SassDoc default theme.

Member

HugoGiraudel commented Aug 21, 2014

@HugoGiraudel not if it cannot access the nested objects as @valeriangalliat said. It doesn't do me any good if I can't have all the data.

You do have all the data. You just can't serialize it because of circular dependencies. That doesn't prevent you from doing things with the data. Hence the SassDoc default theme.

@alademann

This comment has been minimized.

Show comment
Hide comment
@alademann

alademann Aug 21, 2014

Contributor

Ok - is there an example of how a valid JSON structure could be built by crawling the existing data structure?

Contributor

alademann commented Aug 21, 2014

Ok - is there an example of how a valid JSON structure could be built by crawling the existing data structure?

@HugoGiraudel

This comment has been minimized.

Show comment
Hide comment
@HugoGiraudel

HugoGiraudel Aug 21, 2014

Member

What's your point exactly?

Member

HugoGiraudel commented Aug 21, 2014

What's your point exactly?

@alademann

This comment has been minimized.

Show comment
Hide comment
@alademann

alademann Aug 21, 2014

Contributor

I'm not trying to make a point, I'm trying to understand how to get a "complete" snapshot of the sassdoc data in a format that I can use. If thats not something that will be provided I guess I'll have to live with it and parse within the parse provided by sassdoc api - just seems redundant and fragile.

Contributor

alademann commented Aug 21, 2014

I'm not trying to make a point, I'm trying to understand how to get a "complete" snapshot of the sassdoc data in a format that I can use. If thats not something that will be provided I guess I'll have to live with it and parse within the parse provided by sassdoc api - just seems redundant and fragile.

@valeriangalliat

This comment has been minimized.

Show comment
Hide comment
@valeriangalliat

valeriangalliat Aug 21, 2014

Member

It's because of the usedBy keys (at least). The usedBy array contain circular references to other items of the same global object.

If you want something serializable, you have to remove (or convert at your convenience) all the usedBy arrays.

Member

valeriangalliat commented Aug 21, 2014

It's because of the usedBy keys (at least). The usedBy array contain circular references to other items of the same global object.

If you want something serializable, you have to remove (or convert at your convenience) all the usedBy arrays.

@HugoGiraudel

This comment has been minimized.

Show comment
Hide comment
@HugoGiraudel

HugoGiraudel Aug 21, 2014

Member

Don't be so rushy Aaron. We are trying to understand the problem here, and to see if there should be something done in order to fix it. I'd like to have @FWeinb's opinion on this.

Member

HugoGiraudel commented Aug 21, 2014

Don't be so rushy Aaron. We are trying to understand the problem here, and to see if there should be something done in order to fix it. I'd like to have @FWeinb's opinion on this.

@alademann

This comment has been minimized.

Show comment
Hide comment
@alademann

alademann Aug 21, 2014

Contributor

Thanks @valeriangalliat - i'll let you know if I have any success serializing.

Contributor

alademann commented Aug 21, 2014

Thanks @valeriangalliat - i'll let you know if I have any success serializing.

@alademann

This comment has been minimized.

Show comment
Hide comment
@alademann

alademann Aug 21, 2014

Contributor

Sorry @HugoGiraudel - didn't mean to come off that way.

Contributor

alademann commented Aug 21, 2014

Sorry @HugoGiraudel - didn't mean to come off that way.

@alademann

This comment has been minimized.

Show comment
Hide comment
@alademann

alademann Aug 21, 2014

Contributor

My end-goal here is... to gather the sassdoc data into a JSON file so that I can build sassdoc templates using Jekyll's datafiles setup

Contributor

alademann commented Aug 21, 2014

My end-goal here is... to gather the sassdoc data into a JSON file so that I can build sassdoc templates using Jekyll's datafiles setup

@HugoGiraudel

This comment has been minimized.

Show comment
Hide comment
@HugoGiraudel

HugoGiraudel Aug 21, 2014

Member

Ouh, interesting goal actually.

Member

HugoGiraudel commented Aug 21, 2014

Ouh, interesting goal actually.

@FWeinb

This comment has been minimized.

Show comment
Hide comment
@FWeinb

FWeinb Aug 21, 2014

Member

You can pass in a function as a second argument to JSON.stringify to handle the circular structures. I will create a example.

Member

FWeinb commented Aug 21, 2014

You can pass in a function as a second argument to JSON.stringify to handle the circular structures. I will create a example.

@FWeinb

This comment has been minimized.

Show comment
Hide comment
@FWeinb

FWeinb Aug 21, 2014

Member

We could also use circular-json

var CircularJSON = require('circular-json');
var sassdoc = require('sassdoc');

sassdoc
.parse('./sass')
.then(function (items) {
    console.log(CircularJSON.stringify(items));
});

I don't know if the loading of data in Jekyll can be plugged but that would be needed to get it to work.
(You need to use CircularJSON.parse() to get the correct data back)

Member

FWeinb commented Aug 21, 2014

We could also use circular-json

var CircularJSON = require('circular-json');
var sassdoc = require('sassdoc');

sassdoc
.parse('./sass')
.then(function (items) {
    console.log(CircularJSON.stringify(items));
});

I don't know if the loading of data in Jekyll can be plugged but that would be needed to get it to work.
(You need to use CircularJSON.parse() to get the correct data back)

@valeriangalliat

This comment has been minimized.

Show comment
Hide comment
@valeriangalliat

valeriangalliat Aug 21, 2014

Member

Another solution is:

function flattenItem(item) {
  return {type: item.context.type, name: item.context.name};
}

function flattenItems(items) {
  for (var type in items) {
    for (var name in items[type]) {
      if ('usedBy' in items[type][name]) {
        items[type][name].usedBy = items[type][name].usedBy.map(flattenItem);
      }

      if ('requires' in items[type][name]) {
        items[type][name].requires = items[type][name].requires
          .filter(function (item) {return !item.external;})
          .map(function (item) {return item.item;})
          .map(flattenItem);
      }
    }
  }

  return items;
}

require('sassdoc').parse('./sass').then(function (items) {
  console.log(JSON.stringify(flattenItems(items)));
}, function (err) {
  console.error(err);
});

Also I understood why you didn't get any error before: the then method will catch any exception to pass it to the next promise error handler. Since we're using Q, you can use the done method that will throw any remaining exception.

Member

valeriangalliat commented Aug 21, 2014

Another solution is:

function flattenItem(item) {
  return {type: item.context.type, name: item.context.name};
}

function flattenItems(items) {
  for (var type in items) {
    for (var name in items[type]) {
      if ('usedBy' in items[type][name]) {
        items[type][name].usedBy = items[type][name].usedBy.map(flattenItem);
      }

      if ('requires' in items[type][name]) {
        items[type][name].requires = items[type][name].requires
          .filter(function (item) {return !item.external;})
          .map(function (item) {return item.item;})
          .map(flattenItem);
      }
    }
  }

  return items;
}

require('sassdoc').parse('./sass').then(function (items) {
  console.log(JSON.stringify(flattenItems(items)));
}, function (err) {
  console.error(err);
});

Also I understood why you didn't get any error before: the then method will catch any exception to pass it to the next promise error handler. Since we're using Q, you can use the done method that will throw any remaining exception.

@alademann

This comment has been minimized.

Show comment
Hide comment
@alademann

alademann Aug 21, 2014

Contributor

Thanks for looking into this everyone! @FWeinb your method worked... @valeriangalliat - I kept getting the error "cannot read context of undefined" or something like that.

Contributor

alademann commented Aug 21, 2014

Thanks for looking into this everyone! @FWeinb your method worked... @valeriangalliat - I kept getting the error "cannot read context of undefined" or something like that.

@valeriangalliat

This comment has been minimized.

Show comment
Hide comment
@valeriangalliat

valeriangalliat Aug 22, 2014

Member

@alademann Oh, that's odd, I tested it without problems on sassdoc-theme-light's scss folder. Anyway, you have @FWeinb solution, and after all, the JSON encoding is supported with the last @FWeinb's PR. We just need to fix the tests and it will be merged. :)

Member

valeriangalliat commented Aug 22, 2014

@alademann Oh, that's odd, I tested it without problems on sassdoc-theme-light's scss folder. Anyway, you have @FWeinb solution, and after all, the JSON encoding is supported with the last @FWeinb's PR. We just need to fix the tests and it will be merged. :)

@HugoGiraudel HugoGiraudel added this to the 1.5 milestone Aug 23, 2014

valeriangalliat added a commit that referenced this issue Aug 25, 2014

@valeriangalliat

This comment has been minimized.

Show comment
Hide comment
@valeriangalliat

valeriangalliat Aug 25, 2014

Member

There's now a toJSON method in the develop branch, for requires and usedBy objects, to be able to JSON.stringify the data without having circular dependencies.

Member

valeriangalliat commented Aug 25, 2014

There's now a toJSON method in the develop branch, for requires and usedBy objects, to be able to JSON.stringify the data without having circular dependencies.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment