Skip to content

Add --stream flag to ls/grep for SAX-style incremental XML output #4

@ronak-create

Description

@ronak-create

Description

When listing directories with thousands of files or running recursive grep on large codebases, aict currently buffers the entire result set in memory before writing XML. This can cause high memory usage and delayed output for the consumer. Adding a --stream flag would emit XML elements incrementally as they're discovered, similar to SAX-style event streaming.

Current Behavior

$ aict ls /usr/lib --xml
# Buffers all entries, then writes the complete <ls>...</ls> document at once

Proposed Behavior

$ aict ls /usr/lib --stream --xml
# Writes <ls ...> opening tag immediately
# Writes each <file .../> element as it's discovered
# Writes </ls> closing tag when done

Implementation Notes

  • Use xml.NewEncoder(writer) instead of xml.Marshal to stream elements
  • Open the root element with its attributes before processing begins
  • Flush after each element so the consumer receives data incrementally
  • Close the root element on completion or error
  • For error cases mid-stream, emit an <error> element before closing the root tag
  • Apply to ls, grep, and find initially (the tools most likely to produce large result sets)

Example Streaming Output

<ls timestamp="1234567890" total_entries="unknown" path="/usr/lib" absolute="/usr/lib" stream="true">
  <file name="libc.so" .../>
  <file name="libm.so" .../>
  <file name="libpthread.so" .../>
  <!-- elements appear in real-time as files are discovered -->
  <error code="13" msg="permission denied" path="/usr/lib/secret"/>
</ls>

Note: total_entries would be unknown in stream mode since we don't know the count upfront.

Acceptance Criteria

  • --stream flag available on ls, grep, and find
  • First <tool> opening tag is written before any processing begins
  • Each result element is flushed to stdout immediately
  • Root element is properly closed on success and error
  • Memory usage remains constant regardless of result set size
  • --stream is incompatible with --json (JSON requires knowing the array length upfront) — either error or fall back to buffered mode
  • Tests verify: streaming output, error mid-stream, empty results with stream flag

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions