New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
yq v4 is super slow compared to the blazing fast yq v3 #717
Comments
Looking into it - I can make one simple recommendation (and I'll update the docs too), right now with that validate command it will still re-encode the (valid) yaml and pipe it to /dev/null. If you instead run:
Then it will still parse the file (and fail if it's bad) but it won't bother to re-encode it. Depending on how big the files are this should should a good improvement for at least the validation. |
thank you so much for the recommendation @mikefarah. I'm always looking to improve my code. I replaced the validation command with the one you provided. I don't feel any speed improvement, though. The files I parse are relatively small I guess. The longest one is 246 lines, the smallest one is 16 lines. If I run the loop with only the validation command, it takes the same 5s to execute |
I am experiencing orders of magnitude of slowness yq v4 vs v3: V4
V3
Same strace on v4 yields similar output as above plus hendreds of thousands of mutex waiting like there is a shared resource it is waiting to lock on.
|
strange - and that's running with no paramaters? |
Aha I think I found the issue - it's the lexer. |
Give 4.6.1 a try - and let me know how it goes :) |
Much improved! Your fix brought the time under what I was getting in v3
|
Describe the bug
I have a fish shell script that loops over 38 files and executes yq 2 times at each iteration
with yq@3.4.1 it takes ~1 second to execute maybe less.
with yq@4.5.0 it takes 11s
Version of yq: 4.5.0
Operating system: mac 11.2
Installed via: homebrew
Input Yaml
It's docker compose files with approximately 2 services per file, each service defines traefik labels
Example of the kind of files I parse:
https://github.com/gkoerk/docker-swarm-cookbook/blob/master/plex/plex.yml
Command
The commands with yq3:
The commands with yq4:
Actual behavior
The script is slow and freezes the shell for 11s
Expected behavior
The script should be fast as before
Additional context
The full script for version 4.
to test this script, you could
I tested to remove either one of these commands. If I let only
yq e "$file" &> /dev/null
the script takes 5 seconds.If I let only
yq e '.services | keys | .[]' "$file"
, the script executes in 6 seconds.The text was updated successfully, but these errors were encountered: