Osquerybeat: MFT File Table (Windows)

[brian-mckinney]

Proposed commit message

Adds a new elastic_ntfs_file table to the osquery Elastic extension. The table reads file and directory metadata directly from the NTFS $MFT (Master File Table) on Windows, bypassing the filesystem API. This makes it
useful for forensic and threat-hunting queries where standard API access is unreliable — for example, locating files with suspicious timestamps, detecting MFT record reuse via sequence numbers, or enumerating files on
a potentially compromised volume.

● The table supports three query modes:

Mode	Required constraints	Description
Path lookup	path	Resolves a single file or directory by its full Windows path
Inode lookup	inode, drive	Looks up a file directly by MFT record number
Directory scan	directory, filename GLOB	Lists direct children of a directory matching a filename pattern

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works. Where relevant, I have used the stresstest.sh script to run them under stress conditions and race detector to verify their stability.
• I have added an entry in ./changelog/fragments using the changelog tool.

Disruptive User Impact

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

[github-actions]

🤖 GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)
• /test : Run the Buildkite pipeline.

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @brian-mckinney? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[github-actions]

TL;DR

All 5 failing Windows jobs are hitting the same compile-time code bug in x-pack/osquerybeat (*Volume no longer has ntfsContext, but call sites still use it). Update those call sites to use the new lazy session API before rerunning CI.

Remediation

• Replace v.ntfsContext() / volumeInfo.ntfsContext() usages in x-pack/osquerybeat/ext/osquery-extension/pkg/ntfs/file.go and .../parser.go with the current ntfsSession flow (e.g. get session via v.ntfsSession(), then use session.Context() where *parser.NTFSContext is needed).
• Re-run the Windows osquerybeat build/unit stage after updating those references (mage build unitTest under x-pack/osquerybeat).

Investigation details

Root Cause

The Volume type defines ntfsSession func() (*NTFSSession, error) and no ntfsContext field/method (x-pack/osquerybeat/ext/osquery-extension/pkg/ntfs/volume.go:50-62), but build inputs still call ntfsContext:

• ext/osquery-extension/pkg/ntfs/file.go:46 and :155
• ext/osquery-extension/pkg/ntfs/parser.go:53, :95, :112, :124

This mismatch causes Go compilation to fail before tests start.

Evidence

• Build: https://buildkite.com/elastic/beats/builds/45896
• Job/step: :windows: x-pack/osquerybeat: Win 2016/2019/2022/10/11 Unit Tests (all fail identically)
• Key log excerpt:

# github.com/elastic/beats/v7/x-pack/osquerybeat/ext/osquery-extension/pkg/ntfs
ext\osquery-extension\pkg\ntfs\file.go:46:27: f.volume.ntfsContext undefined (type *Volume has no field or method ntfsContext)
ext\osquery-extension\pkg\ntfs\file.go:155:29: volumeInfo.ntfsContext undefined (type *Volume has no field or method ntfsContext)
ext\osquery-extension\pkg\ntfs\parser.go:53:20: v.ntfsContext undefined (type *Volume has no field or method ntfsContext)
...

Verification

• Not run locally in this detective pass (analysis based on Buildkite logs + repository source inspection).

Follow-up

If helpful, I can map each failing call site to the exact replacement expression (ntfsSession + Context()/RawReader()) in a follow-up pass.

Note

🔒 Integrity filter blocked 4 items

The following items were blocked because they don't meet the GitHub integrity level.

• #50641 pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Osquerybeat: MFT File Table (Windows) #50641 pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #50641 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Osquerybeat: MFT File Table (Windows) #50641 issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[infra-vault-gh-plugin-prod]

Pinging @elastic/sec-windows-platform (Team:Security-Windows Platform)

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Enterprise

Run ID: 1c8cff8f-c901-4940-bb86-ec50219303e7

📥 Commits

Reviewing files that changed from the base of the PR and between 3dafd98 and 442245c.

📒 Files selected for processing (3)

• x-pack/osquerybeat/ext/osquery-extension/pkg/ntfs/file.go
• x-pack/osquerybeat/ext/osquery-extension/pkg/ntfs/parser.go
• x-pack/osquerybeat/ext/osquery-extension/specs/elastic_ntfs_file.yaml

🚧 Files skipped from review as they are similar to previous changes (3)

• x-pack/osquerybeat/ext/osquery-extension/specs/elastic_ntfs_file.yaml
• x-pack/osquerybeat/ext/osquery-extension/pkg/ntfs/parser.go
• x-pack/osquerybeat/ext/osquery-extension/pkg/ntfs/file.go

---

📝 Walkthrough

Walkthrough

This PR adds a new osquery table elastic_ntfs_file that surfaces Windows NTFS Master File Table ($MFT) metadata. The implementation includes a table specification defining the schema, documentation with examples, a file node model (fileNode) that materializes MFT entries into queryable results, volume context integration, and query-driven lookup methods supporting inode lookup, exact path resolution, and directory enumeration with glob pattern matching. Supporting changes include a filter utility function (GetColumnConstraints) and a pointer-type reflection fix in GetValueByOsqueryTag.

Possibly related PRs

• elastic/beats#50140: Earlier PR that introduced NTFS volume/partition table infrastructure that this PR extends with file-level MFT querying.

🚥 Pre-merge checks | ✅ 2

✅ Passed checks (2 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

• 🛠️ Update Documentation

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@x-pack/osquerybeat/ext/osquery-extension/pkg/ntfs/file.go`:
- Around line 185-197: NewFileNode currently allows a nil volumeInfo which can
cause downstream nil-pointer dereferences in methods like BuildFullPath and
Materialize; update NewFileNode (constructor for fileNode) to validate
volumeInfo and return a non-nil error (e.g., "invalid volumeInfo provided") when
volumeInfo is nil, similar to the existing mftEntry check, so callers cannot
create a fileNode with a nil volume reference.

In `@x-pack/osquerybeat/ext/osquery-extension/pkg/ntfs/parser.go`:
- Around line 72-75: The current loop swallows predicate errors by logging
(log.Errorf("predicate failed for %q: %v", name, err)) and continuing, which
hides malformed glob errors; change the behavior in the predicate evaluation
(the block referencing name, err, log.Errorf and continue) to return the error
to the caller instead of continuing so directory queries fail fast on invalid
globs, and make the same change for the second occurrence around the predicate
handling at lines noted (the block around the second log.Errorf/continue).
- Around line 33-36: The check in explodePath that validates the drive-letter
prefix is case-sensitive and rejects valid paths like "c:\..."; update the
prefix comparison to be case-insensitive by normalizing both sides (e.g.,
strings.ToLower or strings.ToUpper) or using strings.EqualFold when comparing
the drive-letter. Specifically, replace the existing !strings.HasPrefix(p,
v.DriveLetter) check with a case-insensitive variant such as
!strings.HasPrefix(strings.ToLower(p), strings.ToLower(v.DriveLetter)) (or
compare the first character using strings.EqualFold on p[:len(v.DriveLetter)]
and v.DriveLetter) so paths with different letter casing pass validation.

In `@x-pack/osquerybeat/ext/osquery-extension/specs/elastic_ntfs_file.yaml`:
- Line 1: The file elastic_ntfs_file.yaml contains CRLF line endings which cause
yamllint failures; convert its line endings to LF (Unix-style) to unblock CI by
normalizing the file (e.g., run a dos2unix conversion or update the editor/git
settings so the committed file uses \n), then re-commit the updated
elastic_ntfs_file.yaml; also ensure repository/IDE .gitattributes or
core.autocrlf is configured to prevent future CRLF commits.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Enterprise

Run ID: fb68cf86-48b7-408d-97ce-621809399b94

📥 Commits

Reviewing files that changed from the base of the PR and between c6faf0c and 3dafd98.

⛔ Files ignored due to path filters (2)

• x-pack/osquerybeat/ext/osquery-extension/pkg/tables/generated/ntfs/elastic_ntfs_file/elastic_ntfs_file.go is excluded by !**/generated/**
• x-pack/osquerybeat/ext/osquery-extension/pkg/tables/generated/registry_windows.go is excluded by !**/generated/**

📒 Files selected for processing (10)

• changelog/fragments/1779917047-add-mft-file-table-to-osquerybeat.yaml
• x-pack/osquerybeat/ext/osquery-extension/README.md
• x-pack/osquerybeat/ext/osquery-extension/docs/tables/elastic_ntfs_file.md
• x-pack/osquerybeat/ext/osquery-extension/pkg/filters/filters.go
• x-pack/osquerybeat/ext/osquery-extension/pkg/ntfs/file.go
• x-pack/osquerybeat/ext/osquery-extension/pkg/ntfs/file_test.go
• x-pack/osquerybeat/ext/osquery-extension/pkg/ntfs/parser.go
• x-pack/osquerybeat/ext/osquery-extension/pkg/ntfs/parser_test.go
• x-pack/osquerybeat/ext/osquery-extension/pkg/ntfs/volume.go
• x-pack/osquerybeat/ext/osquery-extension/specs/elastic_ntfs_file.yaml