support yaml input/output in addition to json

[underrun]

i know its a big request - but you never know unless you ask right?

[nicowilliams]

The plan is to have multiple parser options at some point. But only JSON data types (null, boolean, string, number, array, object) will ever be supported.

[underrun]

i suppose that means limits on object keys ala json as well even tho yaml supports just about anything as a key in a mapping?

[nicowilliams]

Correct. jq internally assumes the JSON (not JavaScript) data model. No
NaNs, no infinities, no non-string keys, no difference between tuples and
arrays, only null, booleans, numbers, strings, arrays, and objects.

Changing this would require a ton of work and might require deep changes to
the language. YAML being a superset of JSON, is kinda out, but a subset of
YAML could be OK.

[pkoppstein]

@nicwilliams wrote:

The plan is to have multiple parser options at some point.

One feature request I'd like to make if it isn't already covered by an existing "issue" is what I'll call the "Farewell to awk" flag -- an option to read in a UTF8 file as a sequence of (JSON) strings, one string per line. Is that covered elsewhere?

[nicowilliams]

@pkoppstein There's already raw input and raw output options.

[pkoppstein]

@nicowilliams - Thanks!

[stedolan]

This would definitely be useful. Output is straightforward enough, and input could be done if limited to that subset of YAML that's equivalent to JSON. The JSON-equivalent subset of YAML is a useful language for configuration files and the like (Full YAML is scarily complicated).

Values in jq are always acyclic, so YAML's recursion won't be supported.

[nicowilliams]

@stedolan My plan is to have multiple parsers/encoders accessible from the I/O builtins.

I want parsers for:

• YAML subset (per-this issue and your comment)
• streaming JSON (output pairs of path and leaf value)

[abesto]

Is this something you're planning to work on in the near future? If not, is it something that's feasible for a newcomer to the project to work on?

[nicowilliams]

I should add that while supporting type annotations on input is not
unreasonable, jq will not support "date" or other such types internally,
therefore also not on output (though an output module could use schema to
insert type annotations).

Ideally all such parsers and encoders could be written in jq itself. We'll
need a byte at a time raw input mode for that :(

[nicowilliams]

I should also add that full YAML will never be supported by jq, for obvious reasons :)

[jmdcal]

I found this https://github.com/preaction/ETL-Yertl a perl module that is similar for yaml

[jonseymour]

For those of you who would like to use jq filters on YAML documents, I have written a simple wrapper script that transforms YAML to JSON with a simple python filter, invokes jq on the JSON and then transforms the JSON back to YAML with python.

The script is implemented in bash and will use a locally installed version of jq and python if they are on the path or delegate to a docker container containing the same otherwise.

See http://github.com/wildducktheories/y2j for more details and let me know if you have any feedback.

[underrun]

Transforming yaml to json will lose something - references... knowing something is the "same" object matters for some applications.

[dtolnay]

@jonseymour Awesome! I added a link to y2j in the wiki.

[jonseymour]

@underrun Understood. I have updated the LIMITATIONS section of the README to add further clarification of this point.
@dtolnay - thanks!

[SamuelMarks]

Hmm, there's also this one: https://github.com/abesto/yq

No commits for 2 years then one a month ago

[fadado]

I wrote my own jq wrapper supporting YAML input and output. To use it clone the repository jqt and run make install. The yq script uses Bash and Python with the PyYAML module. The script tries to imitate at maximum the jq command line interface, showing help for example and not requiring redirection of input:

$ yq '.store.book[2]' data/store.yaml
author: Herman Melville
category: fiction
isbn: 0-553-21311-3
price: 8.99
title: Moby Dick

There are also some enhancements, like outputing directly JSON:

$ yq --json -c '.store.book[2]' data/store.yaml
{"category":"fiction","price":8.99,"author":"Herman Melville","isbn":"0-553-21311-3","title":"Moby Dick"}

This is yq on screen help:

$ yq --help
yq - commandline YAML processor
Usage: yq [options] <jq filter> [file]

    yq is a wrapper to jq for processing YAML input, applying the given
    filter to it YAML text input and producing the filter's results as
    YAML or JSON on standard output.

    The options available are yq specific and also from jq. The yq
    options are:
     -h     Show this help
     -J     Preserve JSON output format
     -V     Output the jq version

    Some of the jq options include:
     -e     set the exit status code based on the output
     -f     Read filter from the file f
     -s     read (slurp) all inputs into an array; apply filter to it
     -S     sort keys of objects on output
     --arg a v          set variable $a to value v
     --argjson a v      set variable $a to JSON value v
     --slurpfile a f    set variable $a to an array of values read from f
    Not all jq options have sense using yq.

    For more advanced filters see the jq(1) manpage and
    https://stedolan.github.io/jq

JJOR

[runelabs]

@nicowilliams @wtlangford, I am using jq a lot with docker inspect and Docker-in-docker with very lean Alpine Linux images where support for YAML for docker-compose.yml would be very welcome and increase the usefulness of jq manyfold for me and others.

Because it has so few dependencies it is much more preferable to other tools that need Python, Lua, Perl or similar. Keeping Docker images as lean as possible helps encapsulate specialised build environments/images and other usecases with container images for continuous integration/delivery. jq will easily become an integral staple on all Alpine Docker images - especially with YAML support, so we wouldn't need any additional or alternative tools for custom Docker/Snappy toolchains. 🐳 📦

[earonesty]

here's "yq" instead of jq: https://gist.github.com/earonesty/1d7cb531bb8fff8c228b7710126bcc33

[szepeviktor]

@nicowilliams Please advise.

[kislyuk]

FYI, I have been maintaining https://github.com/kislyuk/yq, which is a straightforward wrapper for jq using PyYAML. (It's also published on PyPI so you can pip install yq - this supersedes https://github.com/abesto/yq).

[pkoppstein]

@kislyuk - Thank you! The jq FAQ now includes the https://github.com/kislyuk/yq link.

[szepeviktor]

@pkoppstein Could you point out where the link is?

[pkoppstein]

@szepeviktor - Thanks for paying attention!

[mikemol]

While I don't have a need for YAML input (and YAML input would certainly be lossy), having a YAML output mode would be extremely handy for being able to generate configuration inputs for tools that don't accept JSON input.

[DonBower]

still looking for @yml or --yaml-output option.

[030]

@DonBower Could you give an example what you are looking for? Does this tool help?

[nichtich]

For YAML output it would be enough to to implement a jq module (aka jq function library) that defines function toyaml. The counterpart may be more difficult and create a performance bottleneck.

[fadado]

On Fri, 10 May 2019, 09:23 Jakob Voß, ***@***.***> wrote: For YAML output it would be enough to to implement a jq module (aka jq function library) that defines function toyaml. The counterpart may be more difficult and create a performance bottleneck.

Perhaps this module https://github.com/fadado/JBOL/blob/master/fadado.github.io/json/json.jq implementing `toxml` (wrapped inside `xmldoc`) can be easily adapted to implement `toyaml`. This shows how to use the module: https://github.com/fadado/JBOL/blob/master/bin/jxml.jq https://github.com/fadado/JBOL/blob/master/bin/jxml JJOR

[voiski]

still looking for @yml or --yaml-output option.

@DonBower If you check the discussion, you will see some options using a combination of tools. You can use tdusmann answer like yq r -j file.yq | jq $filter_command | yq - or the wrapper built by kislyuk that does exactly that.

@030 Use upx to make it smaller, because right now the mikefarah/yq solution is not just smarter but also smaller. We don't need power tools for that, jq is insuperable as parser handler, but we need some good solution to have yaml support =)

[nicowilliams]

When I have the time to get back to jq I'll finish up my I/O branch, and maybe we can then add options for input/output formats, then someone could contribute support for YAML and XML.

[voiski]

@nicowilliams As @OnkelTem mentioned, we have a lot of external solutions with muti-os support. So, I don't see it as a big problem to put pressure.

I believe some folks, including myself, are missing the back messages that bring solutions for that, some more simple and others more complex and complete(I missed that one too).

[nichtich]

Here is an incomplete jq snippet to serialize YAML.

[Blacksmoke16]

Not to add another tool to the list, but I recently created https://github.com/Blacksmoke16/oq.

Has the benefits of wrapping jq, being nearly as performant, and is portable via a singular binary.

I wrote a blog post about it thats shows some of its features, as well as some benchmarks. Is still some work that needs to be done but is off to a good start.

[pkoppstein]

@Blacksmoke16 - Very impressive! I'm also impressed that you used Crystal. I've been waiting, evidently in vain, for Version 1.0, and would be interested to know what your experience has been regarding the pluses and minuses of using Crystal for your project.

[Blacksmoke16]

@pkoppstein It's been quite positive. The syntax and the performance come together quite nicely to make for a nice experience. The other benefits of a single binary and quite extensive standard library made making this pretty easy.

I'd say the main downside currently is you can run into some compiler bugs every now and then which force you to find workarounds until they get fixed. As long as you keep an eye on the changelog, breaking changes haven't been too bad for me.

Also since it's relatively new, library support is not as mature as say Ruby; I find myself spending most of my time making PRs for the libraries I use. However, this project in particular went pretty smooth as it's not too complex, or relying upon fancier features like generics or macros.

If you want to chat more about it feel free to message me on Discord Blacksmoke16#0016.

[tpo]

I created pretty small static binaries for this need at https://github.com/gsf/yaml2json/releases.

It's funny how 1.8M can be considered "pretty small" these days.

[mikemol]

I now routinely use yj to handle translation from YAML to JSON prior to feeding into jq. I selected that tool since it was stupid-simple, and could also handle HCL and TOML.

[OnkelTem]

@mikemol could you please share a link to it?

[mikemol]

@OnkelTem https://github.com/sclevine/yj

[nichtich]

Many tools exist to convert between YAML and JSON but all require additional runtime environments, methods of installation, usage etc. Having jq as single binary with support of several operating system package managers would be most convenient. But I understand if this will not be implemented.

[pauldraper]

The other issue is that YAML allows comments and tools must encode these if they are desired to be kept.

Of course, jq would have the same issue....unless it added core comment support (e.g. for json5).

[captain-kark]

@nicowilliams I motion to take this issue off of life support, as much as I would love to be able to keep everything in one project, the "j" in jq does a fine job for what it is.

[nichtich]

For those still interesting in support of YAML parsing into jq, have a look at https://github.com/biojppm/rapidyaml#comparison-with-yaml-cpp. In contrast to converters mentioned before, it is implemented in C++ so it may be easier to build into jq source code. I'd still take into consideration support of parsing extensions of JSON if the underlying parsing engine does it support anyway.

[SleepyBrett]

I am sad that this issue hasn't had discussion in two years. JSON is a strict subset of yaml, it makes a certain amount of sense just to use a yaml parser on the front end and take all comers. Give me a flag for output in yaml instead of a default json and i'd be super happy.

If there is a poll going I could care less about comments. 99% of the time I use jq I'm transforming json to pass into some other system like kubernetes and none of them care about comments.

[nichtich]

I also use jq to validate JSON, so I'd argue against changing the default input format to accept anything more than sequences of valid JSON documents. YAML output is a different use, it could be supported like existing format strings and escaping @csv, @tsv.... if someone is willing to implement it.

[SleepyBrett]

@nichtich JSON is a subset of yaml. A full json document is 100% valid yaml. Replacing the underlying parsing engine with a yaml parser would not effect you at all.

[Nothing4You]

@SleepyBrett except when you want to stricly accept json but not yaml.