cloud-init: upload only the last 1 MB of each log

[alexey-milovidov]

Summary

• Replace curl ... --data-binary @log / ... @/var/log/cloud-init-output.log with tail -c 1000000 | curl --data-binary @- so we send at most 1 MB per file.
• Systems that build from source (Hyrise, WarehousePG, Spark variants) or that apt-install build-essential produce cloud-init logs of hundreds of MB. The sink enforces a per-row size cap; the unfiltered push failed, and we lost the entire log — including the diagnostic tail we actually use for triage.
• 1 MB tail is enough to capture the bench output, any tracebacks, and the final readiness / timing lines.

Test plan

• Trigger a benchmark on any system whose cloud-init log is known to be small (e.g. clickhouse) and confirm the row lands in sink.data and contains the expected tail.
• Trigger a benchmark on a system known to produce a multi-100MB cloud-init log (e.g. warehousepg, which compiles from source) and confirm:
  • both rows are inserted (per-run log + cloud-init tail),
  • the cloud-init row is ≤ 1 MB,
  • the bottom of the cloud-init row contains the gpinitsystem completion / failure lines.

🤖 Generated with Claude Code