Skip to content

Commit cc20165

Browse files
Copilotlaeubi
Copilot
and
laeubi
authored
Add test suite demonstrating missing transparent HTTP compression support (#15)
* Initial plan * Add HTTP compression tests and documentation - Add JavaHttpClientCompressionTest demonstrating compression limitations - Enhance NettyHttp2Server with gzip compression support - Create HTTP_COMPRESSION.md with detailed analysis and proposal Co-authored-by: laeubi <1331477+laeubi@users.noreply.github.com> * Add GitHub issue summary and update README with compression tests - Create GITHUB_ISSUE_SUMMARY.md for JDK enhancement request - Update README.md to document compression test scenarios Co-authored-by: laeubi <1331477+laeubi@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: laeubi <1331477+laeubi@users.noreply.github.com>
1 parent 5a9eced commit cc20165

File tree

5 files changed

+701
-7
lines changed

5 files changed

+701
-7
lines changed

‎GITHUB_ISSUE_SUMMARY.md‎

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
# GitHub Issue Summary: Add Transparent HTTP Compression Support to Java HTTP Client
2+
3+
## Title
4+
Add transparent HTTP compression support to java.net.http.HttpClient
5+
6+
## Component
7+
core-libs/java.net
8+
9+
## Type
10+
Enhancement
11+
12+
## Summary
13+
14+
The Java HTTP Client (java.net.http.HttpClient) does not support transparent HTTP compression, requiring applications to manually implement both request header management and response decompression. This enhancement request proposes adding automatic compression support similar to other modern HTTP clients.
15+
16+
## Problem
17+
18+
HTTP compression is a fundamental HTTP feature providing significant benefits:
19+
- Reduced bandwidth usage (critical for mobile connections)
20+
- Faster request completion (allowing servers to handle more concurrent connections)
21+
- Better cache utilization (proxies often cache compressed content)
22+
- Improved security (fewer bytes to encrypt/decrypt)
23+
24+
**Current limitations demonstrated by test suite:**
25+
26+
1. The HttpClient does NOT automatically add `Accept-Encoding` header to requests
27+
2. The HttpClient does NOT automatically decompress compressed responses
28+
3. Applications must manually implement both sides of compression support
29+
30+
This creates drawbacks:
31+
- Each application must implement compression independently
32+
- Implementation is non-trivial and error-prone
33+
- Compression is often forgotten or incompletely implemented
34+
- Transport-level compression is treated as an application concern
35+
36+
## Proposed Solution
37+
38+
Add transparent compression support with the following behavior:
39+
40+
**Default (Transparent Mode):**
41+
- Client automatically adds `Accept-Encoding: gzip, deflate` when not explicitly set
42+
- Client automatically decompresses responses based on `Content-Encoding` header
43+
- BodyHandlers receive decompressed content transparently
44+
45+
**Opt-Out Mechanism:**
46+
- Applications can set custom `Accept-Encoding` header to override default behavior
47+
- Setting `Accept-Encoding: identity` explicitly disables compression (per RFC 2616)
48+
- Custom Accept-Encoding values disable automatic decompression
49+
50+
## Benefits
51+
52+
- **Zero-code compression**: Applications get compression without any code changes
53+
- **Backward compatible**: Existing applications continue to work unchanged
54+
- **Standards compliant**: Follows RFC 2616 specifications
55+
- **Consistent with industry**: Matches behavior of curl, browsers, and popular HTTP libraries
56+
57+
## Example Code
58+
59+
### Current Behavior (Manual Implementation Required)
60+
```java
61+
// Must manually add header
62+
HttpRequest request = HttpRequest.newBuilder()
63+
.uri(uri)
64+
.header("Accept-Encoding", "gzip")
65+
.GET()
66+
.build();
67+
68+
HttpResponse<byte[]> response = client.send(request, HttpResponse.BodyHandlers.ofByteArray());
69+
70+
// Must manually check and decompress
71+
if ("gzip".equals(response.headers().firstValue("content-encoding").orElse(""))) {
72+
byte[] compressed = response.body();
73+
// ... manual decompression code (15+ lines) ...
74+
}
75+
```
76+
77+
### Proposed Behavior (Transparent)
78+
```java
79+
// No compression-specific code needed
80+
HttpRequest request = HttpRequest.newBuilder()
81+
.uri(uri)
82+
.GET()
83+
.build();
84+
85+
HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());
86+
String content = response.body(); // Already decompressed!
87+
```
88+
89+
## Supporting Materials
90+
91+
- **Test Suite**: `JavaHttpClientCompressionTest` demonstrates current limitations (5 tests, all passing)
92+
- **Documentation**: `HTTP_COMPRESSION.md` provides detailed technical specification
93+
- **Test Server**: Enhanced `NettyHttp2Server` supports compression for testing
94+
95+
All tests validate that:
96+
- HttpClient does NOT add Accept-Encoding automatically (HTTP and HTTPS)
97+
- HttpClient does NOT decompress responses automatically
98+
- Manual implementation is required for compression support
99+
100+
## References
101+
102+
- [RFC 2616 Section 14.3 - Accept-Encoding](https://datatracker.ietf.org/doc/html/rfc2616#section-14.3)
103+
- [RFC 2616 Section 14.41 - Content-Encoding](https://datatracker.ietf.org/doc/html/rfc2616#section-14.41)
104+
- [Wikipedia - HTTP compression](https://en.wikipedia.org/wiki/HTTP_compression)
105+
106+
## Testing
107+
108+
The enhancement should include:
109+
- Unit tests for gzip and deflate compression/decompression
110+
- Integration tests for transparent mode behavior
111+
- Tests for opt-out mechanism (custom Accept-Encoding headers)
112+
- Tests for both HTTP/1.1 and HTTP/2
113+
- Tests for both HTTP and HTTPS protocols
114+
115+
## Priority Rationale
116+
117+
While HTTP compression is an important feature, applications can currently implement it manually (albeit with significant effort). This enhancement would primarily improve developer experience and reduce boilerplate code. Suggested priority: P4 (Nice to have).

‎HTTP_COMPRESSION.md‎

Lines changed: 245 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,245 @@
1+
# HTTP Compression Support in Java HTTP Client
2+
3+
## Problem Statement
4+
5+
[HTTP compression](https://en.wikipedia.org/wiki/HTTP_compression) is a fundamental capability in modern HTTP communication that provides significant benefits:
6+
7+
1. **Reduced Bandwidth Usage**: Compression reduces data transferred over the network, which is crucial for mobile connections and bandwidth-constrained environments
8+
2. **Faster Transfers**: Reduced data size means faster completion of requests, allowing servers to handle more concurrent connections
9+
3. **Better Cache Utilization**: Many proxies cache compressed content. Requesting uncompressed content can lead to cache misses, resulting in higher loads and slower downloads
10+
4. **Improved Security**: Encryption benefits from compression as fewer bytes need to be encrypted/decrypted, and redundancy is reduced in ciphertext
11+
12+
**Currently, the Java HTTP Client does not support transparent HTTP compression out of the box.** This creates several significant drawbacks:
13+
14+
1. Each application must implement compression support independently
15+
2. Implementation is non-trivial, requiring changes to both request and response handling
16+
3. Compression is often forgotten, incompletely implemented, or contains bugs
17+
4. Transport-level compression is treated as an application concern rather than handled transparently
18+
19+
## Current Behavior Demonstrated by Tests
20+
21+
The test suite in `JavaHttpClientCompressionTest` demonstrates the following current limitations:
22+
23+
### 1. No Automatic Accept-Encoding Header
24+
25+
The Java HTTP Client does **NOT** automatically add the `Accept-Encoding` header to requests:
26+
27+
```java
28+
HttpRequest request = HttpRequest.newBuilder()
29+
.uri(uri)
30+
.GET()
31+
.build();
32+
33+
HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());
34+
35+
// Result: Response does NOT contain Content-Encoding header
36+
// The client never requested compression
37+
```
38+
39+
**Test**: `testNoAutomaticAcceptEncodingHeader()` and `testHttpsNoAutomaticAcceptEncodingHeader()`
40+
41+
### 2. No Automatic Decompression
42+
43+
Even when the `Accept-Encoding` header is manually added and the server responds with compressed content, the Java HTTP Client does **NOT** automatically decompress the response:
44+
45+
```java
46+
HttpRequest request = HttpRequest.newBuilder()
47+
.uri(uri)
48+
.header("Accept-Encoding", "gzip")
49+
.GET()
50+
.build();
51+
52+
HttpResponse<byte[]> response = client.send(request, HttpResponse.BodyHandlers.ofByteArray());
53+
54+
// Result: Response body contains raw gzip-compressed bytes
55+
// Application must manually decompress the data
56+
byte[] compressed = response.body();
57+
// compressed[0] == 0x1f, compressed[1] == 0x8b (gzip magic number)
58+
```
59+
60+
**Test**: `testNoAutomaticDecompression()`
61+
62+
### 3. Manual Implementation Required
63+
64+
Applications must implement both sides of compression manually:
65+
66+
```java
67+
// Step 1: Manually add Accept-Encoding header
68+
HttpRequest request = HttpRequest.newBuilder()
69+
.uri(uri)
70+
.header("Accept-Encoding", "gzip")
71+
.GET()
72+
.build();
73+
74+
HttpResponse<byte[]> response = client.send(request, HttpResponse.BodyHandlers.ofByteArray());
75+
76+
// Step 2: Check if response is compressed
77+
String contentEncoding = response.headers().firstValue("content-encoding").orElse("");
78+
79+
// Step 3: Manually decompress if needed
80+
if ("gzip".equals(contentEncoding)) {
81+
byte[] compressed = response.body();
82+
ByteArrayInputStream bais = new ByteArrayInputStream(compressed);
83+
GZIPInputStream gzipIn = new GZIPInputStream(bais);
84+
ByteArrayOutputStream baos = new ByteArrayOutputStream();
85+
86+
byte[] buffer = new byte[1024];
87+
int len;
88+
while ((len = gzipIn.read(buffer)) > 0) {
89+
baos.write(buffer, 0, len);
90+
}
91+
92+
String decompressed = baos.toString("UTF-8");
93+
// Now we have the actual content
94+
}
95+
```
96+
97+
**Test**: `testManualCompressionRequired()` and `testHttpsManualCompression()`
98+
99+
## Proposed Solution
100+
101+
The Java HTTP Client should support **transparent HTTP compression** similar to other modern HTTP client implementations (e.g., curl, browsers, popular libraries in other languages):
102+
103+
### Default Behavior (Transparent Mode)
104+
105+
When no `Accept-Encoding` header is explicitly set by the application:
106+
107+
1. **Request**: The client automatically adds `Accept-Encoding: gzip, deflate` (or similar appropriate default)
108+
2. **Response**: When the server responds with `Content-Encoding: gzip` (or other supported encoding), the client automatically decompresses the content
109+
3. **BodyHandler**: The `BodyHandler` receives already-decompressed content, requiring no special handling from the application
110+
111+
### Example of Proposed Behavior
112+
113+
```java
114+
// Application code - no compression-specific code needed
115+
HttpRequest request = HttpRequest.newBuilder()
116+
.uri(uri)
117+
.GET()
118+
.build();
119+
120+
HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());
121+
122+
// Result: Client automatically:
123+
// - Sent "Accept-Encoding: gzip, deflate"
124+
// - Received compressed response
125+
// - Decompressed the content transparently
126+
// - Returned decompressed string to application
127+
String content = response.body(); // Already decompressed!
128+
```
129+
130+
### Opt-Out Mechanism
131+
132+
Applications should be able to disable transparent compression when needed:
133+
134+
```java
135+
// Option 1: Explicitly request no compression
136+
HttpRequest request = HttpRequest.newBuilder()
137+
.uri(uri)
138+
.header("Accept-Encoding", "identity") // RFC 2616 Section 14.3
139+
.GET()
140+
.build();
141+
142+
// Option 2: Custom compression handling
143+
HttpRequest request = HttpRequest.newBuilder()
144+
.uri(uri)
145+
.header("Accept-Encoding", "br") // Request only Brotli
146+
.GET()
147+
.build();
148+
// When a custom Accept-Encoding is set, client uses it as-is
149+
// and does not perform automatic decompression
150+
```
151+
152+
### Compatibility
153+
154+
The proposed solution is backward compatible:
155+
156+
- Existing applications that don't use compression continue to work (they now get compression for free)
157+
- Applications that manually implement compression can either:
158+
- Remove their manual implementation and benefit from transparent handling
159+
- Continue using manual implementation by setting custom `Accept-Encoding` headers
160+
- The `identity` encoding allows explicit opt-out per RFC 2616
161+
162+
## Technical Specification
163+
164+
### Supported Encodings
165+
166+
Initial implementation should support:
167+
- `gzip` - GZIP compression (RFC 1952)
168+
- `deflate` - DEFLATE compression (RFC 1951)
169+
170+
Future extensions could include:
171+
- `br` - Brotli compression (RFC 7932)
172+
- `zstd` - Zstandard compression
173+
174+
### Request Handling
175+
176+
1. If `Accept-Encoding` header is **not present** in request:
177+
- Add `Accept-Encoding: gzip, deflate` automatically
178+
- Mark request as using "transparent compression mode"
179+
180+
2. If `Accept-Encoding` header **is present** in request:
181+
- Use the provided value as-is
182+
- Disable transparent decompression (application handles it)
183+
184+
### Response Handling
185+
186+
1. Check for `Content-Encoding` header in response
187+
2. If present and request was in "transparent mode":
188+
- Decompress content using appropriate algorithm
189+
- Make decompressed content available to BodyHandler
190+
- Optionally remove or update `Content-Encoding` header in response object
191+
3. If present but request was **not** in "transparent mode":
192+
- Pass response as-is to application (no decompression)
193+
194+
### BodyHandler Integration
195+
196+
Decompression should happen before the `BodyHandler` receives data:
197+
198+
```
199+
Network → HTTP Client → Decompression (if needed) → BodyHandler → Application
200+
```
201+
202+
This ensures all existing `BodyHandler` implementations work without modification.
203+
204+
## References
205+
206+
- [RFC 2616 Section 14.3 - Accept-Encoding](https://datatracker.ietf.org/doc/html/rfc2616#section-14.3)
207+
- [RFC 2616 Section 14.41 - Content-Encoding](https://datatracker.ietf.org/doc/html/rfc2616#section-14.41)
208+
- [RFC 1952 - GZIP file format specification](https://datatracker.ietf.org/doc/html/rfc1952)
209+
- [RFC 1951 - DEFLATE compression specification](https://datatracker.ietf.org/doc/html/rfc1951)
210+
- [RFC 7932 - Brotli Compressed Data Format](https://datatracker.ietf.org/doc/html/rfc7932)
211+
- [Wikipedia - HTTP compression](https://en.wikipedia.org/wiki/HTTP_compression)
212+
213+
## Test Results
214+
215+
The test suite (`JavaHttpClientCompressionTest`) validates:
216+
217+
✅ Java HTTP Client does NOT automatically add `Accept-Encoding` header (HTTP)
218+
✅ Java HTTP Client does NOT automatically add `Accept-Encoding` header (HTTPS)
219+
✅ Java HTTP Client does NOT automatically decompress gzip responses
220+
✅ Manual compression requires both request header and decompression code
221+
✅ HTTPS behaves identically to HTTP regarding compression
222+
223+
All tests demonstrate the current limitation and the need for transparent compression support.
224+
225+
## Recommendation for JDK Enhancement Request
226+
227+
This document and the accompanying test suite (`JavaHttpClientCompressionTest`) should be used as the basis for submitting an enhancement request to the OpenJDK project. The tests provide concrete evidence of the current limitation, and this document proposes a clear, backward-compatible solution that aligns with HTTP standards and modern HTTP client implementations.
228+
229+
### Suggested JDK Issue Summary
230+
231+
**Title**: Add transparent HTTP compression support to java.net.http.HttpClient
232+
233+
**Component**: core-libs/java.net
234+
235+
**Type**: Enhancement
236+
237+
**Priority**: P4 (Nice to have)
238+
239+
**Description**: (See below for GitHub issue summary)
240+
241+
---
242+
243+
## For Maintainers
244+
245+
The test server (`NettyHttp2Server`) has been enhanced to support gzip compression when clients send the `Accept-Encoding: gzip` header. This allows the test suite to validate both HTTP/1.1 and HTTP/2 compression behavior over both HTTP and HTTPS protocols.

0 commit comments

Comments
 (0)