[push] Validation of datapackage.json on data push is not clear

[pwalsh]

Description

I have an invalid datapackage.json. I try to use data push.

I get the message > Error! Unexpected end of JSON input.

I happen to know as a developer that this error is raised from the method that validates the descriptor. Even without fixing the messages that get thrown by our use of JSON Schema validators on our descriptors, the user experience could be greatly improved by showing the user the context of this error.

Example:

> Running Data Package Validation Step on datapackage.json
> Error! Unexpected end of JSON input

Then I would at least know where the error comes from.

[Mikanebu]

@pwalsh Thanks for reporting this! We have a separate command data validate which validates metadata(latest spec) and data against a schema. We planned to include validation before pushing any data. We will work on this on incoming sprint.

[pwalsh]

@Mikanebu I think you must already be validating on data push. Otherwise, why would I get this error?

[pwalsh]

I'm running:

pwalsh:test pwalsh$ data --version
0.5.0

[Mikanebu]

@pwalsh Yes, you are right, it validates json file structure, I meant validation datapackage.json against the latest schema... We will improve the context of this error.

[anuveyatsu]

@pwalsh thanks for the feedback! You're getting this error on reading datapackage.json. We do not validate descriptor on push atm.

Also, can I suggest to use our Desktop app so your CLI tool is always up to date - https://github.com/datahq/data-desktop/releases (it is available only for MacOS right now).

[pwalsh]

@anuveyatsu I know about the desktop app, but personally, I prefer CLI tools and have average to poor experience with Electron apps (Slack etc. are resource hungry). So, for now at least, I really prefer to use the CLI from npm. As I pulled the CLI from npm in the last hour, I assume it is up-to-date?

[anuveyatsu]

@pwalsh we've released a newer version today so I believe NPM is out of date. But I will publish right now so you can re-install.

[AcckiyGerman]

@zelima

[name=Meiran] Can we close this issue #23 as a fixed in datopian/data-cli#231 ?. And saying that we do hangle validation on data validate command. data push only validates datapackage.json against schema

[zelima]

@AcckiyGerman I don't think I understand you.
Do you mean we do not validate data prior push?

[AcckiyGerman]

@zelima please forget this question, we already discussed it, here is the log:

Dima German @AcckiyGerman 12:40
I tried to data push with invalid dp.json (deleted one comma)

user@pc:~/work/datasets/gdp$ data push
> Error! Unexpected string in JSON at position 1225
user@pc:~/work/datasets/gdp$ data validate
> Error! Unexpected string in JSON at position 1225
> Error! Parse error on line 47:
...d OECD"    }  ]  "title": "Country, R
--------------------^
Expecting 'EOF', '}', ',', ']', got 'STRING'

so data push error is not obvious enough I think, data validate error is cool )

Meiran Zhiyenbayev @Mikanebu 12:39
You see, data validate handles it, but data push not
data push handles metadata validation agains specs

Dima German @AcckiyGerman 12:40
What if we add some info for this error? like

user@pc:~/work/datasets/gdp$ data push
> Error! Unexpected string in JSON at position 1225
> You can run 'data validate' to get additional info

???

Anuar Ustayev @anuveyatsu 12:40
@AcckiyGerman yes, probably would be fine

Meiran Zhiyenbayev @Mikanebu 12:40
i like it

Dima German @AcckiyGerman 12:41
Ok, I will do a PR

[AcckiyGerman]

FIXED:
datopian/data-cli#231 and datopian/data-cli#277 are the solution for this issue:

• User will get several suggestions if he got the data push error
• the error itself is a bit more informative