bug： httpx execution cannot be started without closing the standard input pipe

[hktalent]

describe:

When executing an httpx command using exec.Command, I discovered that if the standard input pipe wt is not closed, httpx gets stuck and never starts execution. This prevents streaming and may cause memory overflow issues. This problem does not exist with the tlsx command.

Steps to reproduce:

• Use exec.Command to execute the httpx command and get the standard input pipe wt.
• Write data to wt without closing wt.
• Wait for httpx to execute and output the results.

Expected behavior:

Even if the standard input pipe wt is not closed, httpx should be able to start execution and process the input stream.

Actual behavior:

httpx gets stuck and fails to start execution.

Additional Information:

• Test code: func TestDoCmdNode1(t *testing.T) { ... }
• httpx version: v1.3.7
• Operating system: Darwin VUnderline51pwn.local 23.2.0 Darwin Kernel Version 23.2.0: Wed Nov 15 21:54:10 PST 2023; root:xnu-10002.61.3~2/RELEASE_X86_64 x86_64

test code

var re1 = regexp.MustCompile(` +`)
func TestDoCmdNode1(t *testing.T) {
	a := re1.Split(`httpx -title -websocket -method -server -location -ip  -pipeline -fr -csp-probe -http2 -p 443,80 -nc -silent -td -cname -t 64 -json`, -1)
	Cmd := exec.Command(a[0], a[1:]...)
	var err error
	var wt io.WriteCloser
	var wg sync.WaitGroup
	wg.Add(1)
	Cmd.Stdout = os.Stdout
	go func() {
		defer wg.Done()
		if wt, err = Cmd.StdinPipe(); nil == err {
			wt.Write([]byte("www.163.com\n"))
			/*
					I am not in a hurry to close wt here, there will be more uses in the future
					At this point I hope httpx has started running and output the results
					But we found that if wt is not closed here, http will hang and will never start execution.
				In other words, it will not friendly process and execute the stream from time to time.
				When the stream becomes larger one by one, it will cause memory overflow, the same problem. , tlsx does not have this problem. It can process the stream line by line in a friendly manner without waiting for the stream to be closed.
			*/
			//wt.Close()
		} else {
			fmt.Println(err)
		}
	}()
	wg.Wait()
	Cmd.Start()
	Cmd.Wait()
}

[RamanaReddy0M]

@hktalent httpx already has stream support, try including the -stream option in the test code.

....
a := re1.Split(`httpx -stream -title -websocket -method -server -location -ip  -pipeline -fr -csp-probe -http2 -p 443,80 -nc -silent -td -cname -t 64 -json`, -1)
// existing test code
...

Let me know if this doesn't work for you.

[Mzack9999]

@hktalent as @RamanaReddy0M mentioned for the time being this can be mitigated with -stream modality. In future all tools will be reworked to support continuous stdin/stdout. Since in the example you are using golang, I would suggest also to attempt using httpx as SDK, you can create and use concurrently multiple runners.

[hktalent]

@Mzack9999
I once used projectdiscovery projects as SDKs, but found that there were conflicts in the third-party packages used by these toolsets, and the integration cost based on SDKs was very expensive, and various conflicts, memory leaks and other issues needed to be resolved. Thank you very much anyway

[hktalent]

@RamanaReddy0M thanks