Skip to content

Support bzip2 and xz compressed inputs in LLM file analysis - #70302

Open
guan404ming wants to merge 1 commit into
apache:mainfrom
guan404ming:common-ai-file-analysis-compression
Open

Support bzip2 and xz compressed inputs in LLM file analysis#70302
guan404ming wants to merge 1 commit into
apache:mainfrom
guan404ming:common-ai-file-analysis-compression

Conversation

@guan404ming

Copy link
Copy Markdown
Member

Why

  • LLMFileAnalysisOperator only accepted gzip, so archived logs and exports stored as .log.xz or .csv.bz2 were rejected outright.
  • The path suffixes were already recognised, but any codec other than gzip was refused, forcing users to decompress in a separate task.

How

  • Accept bzip2 and xz alongside gzip for .log, .json, and .csv inputs, using the standard library only.
  • Replace the hardcoded gzip branch with a codec lookup, so all supported codecs share one read path.
  • Keep snappy and zstd rejected, and keep compression unsupported for Parquet, Avro, image, and PDF inputs.

Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 4.8)

Generated-by: Claude Code (Opus 4.8) following the guidelines

@guan404ming
guan404ming force-pushed the common-ai-file-analysis-compression branch from fcd96ae to 45bb2b6 Compare July 23, 2026 14:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant