test: add end-to-end test for CDC chunk compression

[kalbasit]

This commit adds a new test case 'testCDCChunksAreCompressed' to the CDC test suite in pkg/cache/cdc_test.go.
This test verifies that when Content-Defined Chunking (CDC) is enabled, chunks are transparently compressed using ZSTD before being stored in the chunk store.

Key validations:

• Chunks are stored with the expected compression.
• The 'compressed_size' is correctly tracked in the database.
• Data reassembly (GetNar) is transparent and returns the original decompressed content.

This ensures that the recent ZSTD compression features for Local and S3 chunk stores are working correctly in an end-to-end workflow.

During the test, it was discovered that the compressed size was not written to the database. This was fixed.

[kalbasit]

• test: improve and speed up testing #867
• feat: implement ZSTD compression pooling and integrate across project #865
• test: add end-to-end test for CDC chunk compression #864 👈 (View in Graphite)
• feat: add zstd compression to s3 chunk store #863
• feat: add zstd compression to local chunk store #862
• refactor: update PutChunk interface to return compressed_size #861
• feat: add compressed_size column to chunks table for CDC compression tracking #860
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[gemini-code-assist]

Summary of Changes

Hello @kalbasit, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a new end-to-end test case to the CDC test suite. The primary goal is to verify the correct functioning of ZSTD compression for Content-Defined Chunking (CDC) when chunks are stored in local or S3 chunk stores. It ensures that data is compressed effectively, metadata is updated, and the original content can be transparently retrieved.

Highlights

• Compression Verification: Ensures that Content-Defined Chunking (CDC) chunks are transparently compressed using ZSTD before storage.
• Metadata Tracking: Confirms that the 'compressed_size' is accurately tracked in the database for compressed chunks.
• Data Integrity: Validates that data reassembly (GetNar) is transparent and returns the original decompressed content, maintaining data integrity.

Changelog

• pkg/cache/cdc_test.go
  • Added a new test case "testCDCChunksAreCompressed" to validate end-to-end ZSTD compression for CDC chunks.
  • Registered the new test case within the "runCDCTestSuite".

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[dosubot]

Related Documentation

Checked 7 published document(s) in 1 knowledge base(s). No updates required.

^{How did I do? Any feedback?}  

[gemini-code-assist]

Code Review

This pull request adds an end-to-end test for CDC chunk compression. The new test testCDCChunksAreCompressed is a good addition to verify the compression feature. However, the test as implemented is incomplete. It checks for the creation of chunks but fails to verify the core aspect of compression: that the stored chunks are actually smaller than the original data and that the compressed size is tracked correctly in the database. I've provided a suggestion to enhance the test to perform these crucial validations. With this improvement, the test will provide much stronger guarantees about the correctness of the CDC compression feature.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 3.96%. Comparing base (7c5de21) to head (a6f7400).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@          Coverage Diff          @@
##            main    #864   +/-   ##
=====================================
  Coverage   3.96%   3.96%           
=====================================
  Files          6       6           
  Lines        429     429           
=====================================
  Hits          17      17           
  Misses       409     409           
  Partials       3       3           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.