Discard oversized Content-Length and oversized streaming responses during download

[yzqzss]

close: #368

[NGTmeaty]

Code looks good (besides the one request!), but I did just test with a radio stream (no content-length) and the file appears to be recorded to WARC still 🤔

URL: http://rtlberlin.streamabc.net/rtlb-rtldenational-mp3-128-2770113?sABC=685s441s%230%236191718o6s8p33pr344pp16s8s41n2po%23&aw_0_1st.playerid=&amsparams=playerid:;skey:1751073823 (from http://stream.rtlradio.de/rtl-de-national/mp3-128/play.m3u )

Logs:

2025-06-27T21:40:59-04:00 INFO  reactor.go:58       | started component=reactor
2025-06-27T21:40:59-04:00 INFO  preprocessor.go:70  | started component=preprocessor
2025-06-27T21:40:59-04:00 INFO  archiver.go:75      | bucket manager started component=archiver
2025-06-27T21:40:59-04:00 INFO  archiver.go:89      | started component=archiver
2025-06-27T21:40:59-04:00 INFO  postprocessor.go:53 | started component=postprocessor
2025-06-27T21:40:59-04:00 INFO  pipeline.go:130     | starting local queue component=controler.StartPipeline
2025-06-27T21:40:59-04:00 INFO  lq.go:65            | started component=lq
2025-06-27T21:40:59-04:00 INFO  finisher.go:55      | started component=finisher
2025-06-27T21:42:51-04:00 ERROR archiver.go:336     | unable to process body component=archiver.archive worker_id=0 err=Response content-length exceeds configured limit item_id=575ea seed_id=575ea depth=0 hops=0

^C2025-06-27T21:42:59-04:00 INFO  signal.go:27        | received shutdown signal, stopping services... component=controler.signalWatcher
2025-06-27T21:42:59-04:00 INFO  reactor.go:87       | frozen component=reactor
2025-06-27T21:42:59-04:00 INFO  preprocessor.go:85  | stopped component=preprocessor
2025-06-27T21:43:00-04:00 INFO  archiver.go:126     | stopped component=archiver
2025-06-27T21:43:00-04:00 INFO  archiver.go:131     | closed bucket manager component=archiver
2025-06-27T21:43:00-04:00 INFO  postprocessor.go:67 | stopped component=postprocessor
2025-06-27T21:43:00-04:00 INFO  finisher.go:73      | stopped component=finisher
2025-06-27T21:43:01-04:00 INFO  lq.go:89            | stopped component=lq
2025-06-27T21:43:01-04:00 INFO  reactor.go:78       | stopped component=reactor
2025-06-27T21:43:01-04:00 INFO  pipeline.go:206     | done, logs are flushing and will be closed component=controler.stopPipeline

WARC excerpt:

WARC/1.1
WARC-IP-Address: 167.160.93.106
WARC-Record-ID: <urn:uuid:e6f22e29-f52b-48a6-845c-2b52074b7845>
WARC-Block-Digest: sha1:3FRXP5S6TWB23QMRHZ32RVOISVBOMD52
WARC-Date: 2025-06-28T01:40:59.608670178Z
Content-Type: application/http; msgtype=response
WARC-Payload-Digest: sha1:MG55MGUGFVGLDB4NBOYJFY6UOZCNI6D7
WARC-Concurrent-To: <urn:uuid:b973801a-0b55-47ca-9d44-d65b49215031>
WARC-Target-URI: http://rtlberlin.streamabc.net/rtlb-rtldenational-mp3-128-2770113?sABC=685s441s%230%236191718o6s8p33pr344pp16s8s41n2po%23&aw_0_1st.playerid=
Content-Length: 2098495
WARC-Warcinfo-ID: <urn:uuid:75655cf7-4cc6-4706-b4f5-33b7a7b7dd7d>
WARC-Type: response

HTTP/1.1 200 OK
Accept-Ranges: none
Access-Control-Allow-Headers: Origin, Accept, X-Requested-With, Content-Type, Icy-MetaData
Access-Control-Allow-Methods: GET, OPTIONS, SOURCE, PUT, HEAD, STATS
Access-Control-Allow-Origin: *
Cache-Control: no-cache, no-store
Content-Type: audio/mpeg
Date: Sat, 28 Jun 2025 01:41:00 UTC
Expires: Mon, 26 Jul 1997 05:00:00 GMT
Pragma: no-cache
Server: QuantumCast Streamer (Version  SHA:fee09454e42b1065ce924637077fa12f589f037f 25.06.26 07:46:26)
Vary: Origin
X-Quantumcast-Channelkey: rtl-de-national
X-Quantumcast-Listener: 6191718b6f8c33ce344cc16f8f41a2cb
X-Quantumcast-Session: d1fkgaons4ds72p9rmh0
... (actual content follows)

with the following request:

WARC/1.1
WARC-Block-Digest: sha1:MIBTROL5DJDS7Z3HIBUUXUNPTUKEO7RZ
Content-Length: 325
WARC-Warcinfo-ID: <urn:uuid:75655cf7-4cc6-4706-b4f5-33b7a7b7dd7d>
WARC-Type: request
Content-Type: application/http; msgtype=request
WARC-Record-ID: <urn:uuid:b973801a-0b55-47ca-9d44-d65b49215031>
WARC-Concurrent-To: <urn:uuid:e6f22e29-f52b-48a6-845c-2b52074b7845>
WARC-Date: 2025-06-28T01:40:59.608670178Z
WARC-IP-Address: 167.160.93.106
WARC-Target-URI: http://rtlberlin.streamabc.net/rtlb-rtldenational-mp3-128-2770113?sABC=685s441s%230%236191718o6s8p33pr344pp16s8s41n2po%23&aw_0_1st.playerid=

GET /rtlb-rtldenational-mp3-128-2770113?sABC=685s441s%230%236191718o6s8p33pr344pp16s8s41n2po%23&aw_0_1st.playerid= HTTP/1.1
Host: rtlberlin.streamabc.net
User-Agent: Mozilla/5.0 (compatible; archive.org_bot +http://archive.org/details/archive.org_bot) Zeno/f804c4f warc/v0.8.80
Accept-Encoding: gzip
Connection: close

Just to note as well, a large file with Content-Length did not get recorded, so it is partially working!

[yzqzss]

Just to note as well, a large file with Content-Length did not get recorded, so it is partially working!

It should be fixed in internetarchive/gowarc#115. (Bug 1 and Feature 1)

[Copilot]

Pull Request Overview

This PR refactors the configuration and handling of HTTP timeouts and response discards to ensure that oversized Content-Length values and streaming responses exceeding the configured limits are properly discarded. Key changes include:

• Updating configuration fields to use time.Duration for HTTPTimeout and introducing ConnReadDeadline.
• Adding a new hook for discarding responses based on their Content-Length.
• Refactoring body processing to incorporate a custom connection wrapper for improved read deadline handling.

Reviewed Changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
internal/pkg/config/config.go	Updated timeout fields and added a new MaxContentLengthMiB field.
internal/pkg/archiver/warc.go	Adjusted connection deadline handling with new field usage.
internal/pkg/archiver/discard/discarder/warcdiscardstatus/warcdiscardstatus.go	Removed redundant slice length check in WARC discard status hook.
internal/pkg/archiver/discard/discarder/contentlength/contentlength.go	Introduced a new hook to discard responses exceeding a content length limit.
internal/pkg/archiver/discard/discard.go	Updated default discard hooks to include the content length hook if enabled.
internal/pkg/archiver/body.go	Refactored body processing to use the new connection wrapper and removed manual deadline resets.
internal/pkg/archiver/archiver.go	Integrated wrapped connection via channel and handled response discards correctly.
cmd/get.go	Updated command line flags to support new timeout and content length settings.

Comments suppressed due to low confidence (2)

internal/pkg/config/config.go:69

• The field name 'MaxContentLengthMiB' implies the limit is in mebibytes, but the CLI flag in cmd/get.go describes the limit in MB. Consider aligning the unit naming and documentation for clarity.

	MaxContentLengthMiB      int           `mapstructure:"max-content-length"`

cmd/get.go:54

• The flag description uses MB while the configuration field is named with MiB. It would improve clarity to ensure the units are consistent across the codebase.

	getCmd.PersistentFlags().Int("max-content-length", 0, "Max content length in MB to download for a single resource.")

[NGTmeaty]

Looks great! Thanks for getting that issue resolved!