connection reset for response more than 4 MB (introduced in nuclei v3.1.6)

[savushkin-yauheni]

Hi all!

Nuclei version:

from v3.1.6 - to last one

Current Behavior:

nuclei somehow close connection and doesn't read whole response for big responses (more than 4mb in my case)

Expected Behavior:

read whole response as in v3.1.5

Steps To Reproduce:

I created a python web server (added as txt file, rename please to .py and run it):
python9.txt

Use simple template:

id: 200-code

info:
  name: 200 codes
  author: savik
  severity: info

requests:
  - method: GET
    path:
      - "{{BaseURL}}"
    skip-variables-check: true
    matchers:
      - type: status
        status:
          - 200

---

After that:

problematic version v3.1.6 or last one:

go install -v github.com/projectdiscovery/nuclei/v3/cmd/nuclei@v3.1.6
nuclei -duc  -nh -retries 0 -rl 100 -bs 200 -target http://0.0.0.0:2379/debug_mode_nuclei_dump -t template.yaml

We got in our python server log:

GET ex: 753ac718-2ce8-46a7-bff4-b46b3389e98d [Errno 54] Connection reset by peer 

or 

GET ex: 201e2d62-e16e-4057-9d60-5552e7697234 [Errno 32] Broken pipe

---

Let's check v3.1.5:

go install -v github.com/projectdiscovery/nuclei/v3/cmd/nuclei@v3.1.5
nuclei -duc  -nh -retries 0 -rl 100 -bs 200 -target http://0.0.0.0:2379/debug_mode_nuclei_dump -t template.yaml

everything is okay and whole response was read without any errors.

Video

Screen.Recording.2024-02-03.at.14.43.02.mov

Thanks and regards

[ehsandeep]

@savushkin-yauheni it was intensional change as part of #4630 as large response causing memory leaks / higher memory uses!

did you tried using max-size attribute in template options?

[savushkin-yauheni]

Hi @ehsandeep !
Thanks for your quick response. I see, but I tried with max-size and there is still a bug:

id: 200-code

info:
  name: 200 codes
  author: savik
  severity: info

requests:
  - method: GET
    max-size: 1000000000
    path:
      - "{{BaseURL}}"
    skip-variables-check: true
    matchers:
      - type: status
        status:
          - 200

And according to the code, it doesn't consider such option:

	if resp != nil && resp.Body != nil {
		resp.Body = protocolutil.NewLimitResponseBody(resp.Body)
	}

var (
	MaxBodyRead = int64(1 << 22) // 4MB using shift operator
)

func NewLimitResponseBody(body io.ReadCloser) io.ReadCloser {
	return NewLimitResponseBodyWithSize(body, MaxBodyRead)
}
	

What do you think about it? From my point of view max-size should be considered.

Thanks!

[ehsandeep]

@savushkin-yauheni thanks for confirming, are you looking to read / match against partial response (4 MB) / specified size or reading whole response without any limits?

[savushkin-yauheni]

@ehsandeep it's a good question.
I think specified size would be enough, because we don't need to invent some new flags\whatever. Just check max-size and it would be great.