Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sqlite-utils insert --flatten option to flatten nested JSON #310

Closed
simonw opened this issue Aug 9, 2021 · 3 comments
Closed

sqlite-utils insert --flatten option to flatten nested JSON #310

simonw opened this issue Aug 9, 2021 · 3 comments
Labels
cli-tool enhancement New feature or request

Comments

@simonw
Copy link
Owner

simonw commented Aug 9, 2021

I had to do this with a jq recipe today: https://til.simonwillison.net/cloudrun/tailing-cloud-run-request-logs

cat log.json | jq -c '[leaf_paths as $path | {
  "key": $path | join("_"), "value": getpath($path)
}] | from_entries' \
| sqlite-utils insert /tmp/logs.db logs - --nl --alter --batch-size 1

That was to turn something like this:

  {
    "httpRequest": {
      "latency": "0.112114537s",
      "requestMethod": "GET",
      "requestSize": "534",
      "status": 200,
    },
    "insertId": "6111722f000b5b4c4d4071e2",
    "labels": {
      "service": "datasette-io"
    }
  }

Into this instead:

{
    "httpRequest_latency": "0.112114537s",
    "httpRequest_requestMethod": "GET",
    "httpRequest_requestSize": "534",
    "httpRequest_status": 200,
    "insertId": "6111722f000b5b4c4d4071e2",
    "labels_service": "datasette-io"
}

I have to do this often enough that I think it should be an option, --flatten - so I can do this instead:

cat log.json | sqlite-utils insert /tmp/logs.db logs - --flatten
@simonw simonw added cli-tool enhancement New feature or request labels Aug 9, 2021
@simonw simonw closed this as completed in f67327a Aug 9, 2021
@simonw
Copy link
Owner Author

simonw commented Aug 9, 2021

simonw added a commit that referenced this issue Aug 9, 2021
@simonw
Copy link
Owner Author

simonw commented Aug 9, 2021

simonw added a commit that referenced this issue Aug 9, 2021
simonw added a commit that referenced this issue Aug 9, 2021
@rdtq
Copy link

rdtq commented Oct 16, 2021

It would be cool if --flatten worked with sqlite-utils memory as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cli-tool enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants