Skip to content

Use streaming I/O for markdown output instead of loading entire file into memory #4

@deevus

Description

@deevus

Problem

readSymbolMarkdown currently reads the entire markdown cache file into memory before writing it to stdout. This was a workaround for a Zig stdlib bug on macOS where reader.stream() / reader.streamRemaining() fails with EBADF (errno 9).

The root cause is that Zig's std.Io.Writer.sendFile uses fcopyfile on macOS, which doesn't support pipes as the destination fd. When stdout is a pipe (e.g., gdoc Node3D | less), fcopyfile fails.

Current workaround

const content = try file.readToEndAlloc(allocator, 1024 * 1024);
defer allocator.free(content);
try output.writeAll(content);

This works but allocates O(n) memory instead of streaming through a fixed buffer.

Desired solution

Use reader.streamRemaining(output) (or equivalent) to stream file content to the output writer with constant memory usage. This requires either:

  1. A fix upstream in Zig's sendFile / fcopyfile handling for pipe destinations on macOS
  2. A custom streaming loop that bypasses sendFile (read into fixed buffer, write to output, repeat)

Upstream

No existing issue found on codeberg.org/ziglang/zig for this specific bug. Consider filing one.

References

  • src/cache.zig: readSymbolMarkdown function
  • Zig stdlib: std/fs/File.zig sendFile, std/Io/Writer.zig sendFile, std/Io/Reader.zig stream/streamRemaining

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions