feat: dig template function + BoltDB empty-root fix + skip-on-error#140
Merged
jamestexas merged 3 commits intomainfrom Mar 27, 2026
Merged
feat: dig template function + BoltDB empty-root fix + skip-on-error#140jamestexas merged 3 commits intomainfrom
jamestexas merged 3 commits intomainfrom
Conversation
dig: safely navigate nested maps/slices by dot-separated path.
Returns "" if any intermediate key is missing, nil, or out of bounds.
Supports map keys and integer indices for slices.
Enables mixed-format schemas (e.g. vunnel + OSV in one projection).
{{dig "item.Vulnerability.NamespaceName" .}} → "alpine:3.18" or ""
{{dig "item.affected.0.package.ecosystem" .}} → "AlmaLinux:8" or ""
BoltDB materializer:
- Rewritten to use parent_id tree traversal (not path splitting on /)
- Handles node names containing slashes, parens, spaces correctly
- Empty-named root nodes are transparent (children become top-level buckets)
- Self-referencing nodes (id=="" parent_id=="") don't cause infinite recursion
- 7 tests including empty-root and existing tests
Skip-on-error: template render failures in node/file names log WARN and
skip the record instead of aborting the entire ingestion. Enables schemas
that handle mixed-format data sources gracefully.
Build: Taskfile.yml now includes -tags boltdb by default.
There was a problem hiding this comment.
Pull request overview
This PR improves ingestion/materialization robustness for mixed-format vulnerability data by adding a safe dig template helper, making ingestion optionally tolerant of template render failures, and fixing BoltDB materialization edge cases around empty-root nodes and special characters in node names.
Changes:
- Add
dig(path, obj)template function for safe traversal of nested maps/slices in templates. - Update ingestion to skip (instead of error) when a node name template fails to render, enabling mixed-shape records to coexist.
- Rework BoltDB materializer traversal to use
parent_id+name(not path splitting) and add coverage for empty-root node behavior; enableboltdbtag intask build.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| internal/materialize/boltdb.go | Replaces path-splitting with parent/child traversal; adds empty-root handling and recursive materialization. |
| internal/materialize/boltdb_test.go | Adds a regression test for empty-root node schemas using an on-disk SQLite fixture. |
| internal/ingest/engine.go | Adds dig template func and changes name-render failures to warn+skip for mixed-format handling. |
| internal/ingest/dig_test.go | Adds unit tests covering dig behavior across missing keys, arrays, nils, and composition with other funcs. |
| Taskfile.yml | Builds with -tags boltdb by default. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Remove redundant `any` type annotation in dig func (lint fix)
- Add visited set to materializeBoltChildren for cycle detection
- Restrict transparent root passthrough to actual root (parentBucket==nil)
- Guard empty path components in dig ("a..b" no longer returns index 0)
- Add -tags boltdb to test task so boltdb tests run in CI
- Document _root bucket behavior in BoltDBMaterializer doc comment
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Enables mixed-format Trivy projection (vunnel + OSV) via dig template function, fixes BoltDB materializer for empty root nodes and special characters in node names, and adds skip-on-error for graceful mixed-format handling. 7 new tests.