support json output format #294

RamanaReddy0M · 2023-05-15T20:00:43Z

#261 JSONL output format support

ehsandeep

Panic with -j option

./proxify -j

                       _ ___    
   ___  _______ __ __ (_) _/_ __
  / _ \/ __/ _ \\ \ // / _/ // /
 / .__/_/  \___/_\_\/_/_/ \_, / 
/_/                      /___/

		projectdiscovery.io

[INF] Current proxify version 0.0.9 (latest)
[INF] HTTP Proxy Listening on 127.0.0.1:8888
[INF] Socks5 Proxy Listening on 127.0.0.1:10080
[INF] Saving proxify traffic to logs
fatal error: concurrent map writes

goroutine 754 [running]:
github.com/projectdiscovery/proxify/pkg/logger.(*Logger).LogRequest(0xc00031e150, 0xc001037500, {{0xc001681860, 0x10}, 0x0, 0x0, {0xc000850757, 0xe}})
	/Users/geekboy/Github/proxify/pkg/logger/logger.go:149 +0x129
github.com/projectdiscovery/proxify.(*Proxy).ModifyRequest(0xc0001140f0, 0xc001037500)
	/Users/geekboy/Github/proxify/proxy.go:115 +0x2b6
github.com/projectdiscovery/martian/v3.(*Proxy).handleConnectRequest(0xc000536800, 0xc0001a0a80?, 0xc001037500, 0xc000213a20?, 0xc000213a10, {0x25e1bc8?, 0xc00087a848})
	/Users/geekboy/go/pkg/mod/github.com/projectdiscovery/martian/v3@v3.0.0-20230412114616-98e3a0a6994a/proxy.go:392 +0x86
github.com/projectdiscovery/martian/v3.(*Proxy).handle(0xc000536800, 0xc0001a0a00, {0x25e1bc8, 0xc00087a848}, 0xc000213a10)
	/Users/geekboy/go/pkg/mod/github.com/projectdiscovery/martian/v3@v3.0.0-20230412114616-98e3a0a6994a/proxy.go:608 +0x512
github.com/projectdiscovery/martian/v3.(*Proxy).handleLoop(0xc000536800, {0x25e1bc8?, 0xc00087a848?})
	/Users/geekboy/go/pkg/mod/github.com/projectdiscovery/martian/v3@v3.0.0-20230412114616-98e3a0a6994a/proxy.go:313 +0x725
created by github.com/projectdiscovery/martian/v3.(*Proxy).Serve
	/Users/geekboy/go/pkg/mod/github.com/projectdiscovery/martian/v3@v3.0.0-20230412114616-98e3a0a6994a/proxy.go:274 +0x478

goroutine 1 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x2d80d788, 0x72)
	/usr/local/go/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc000536a80?, 0x12d90cb?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc000536a80)
	/usr/local/go/src/internal/poll/fd_unix.go:614 +0x2bd
net.(*netFD).accept(0xc000536a80)
	/usr/local/go/src/net/fd_unix.go:172 +0x35
net.(*TCPListener).accept(0xc000010a68)
	/usr/local/go/src/net/tcpsock_posix.go:148 +0x25
net.(*TCPListener).Accept(0xc000010a68)
	/usr/local/go/src/net/tcpsock.go:297 +0x3d
github.com/armon/go-socks5.(*Server).Serve(0xc0002b6f90, {0x25da0b0, 0xc000010a68})
	/Users/geekboy/go/pkg/mod/github.com/armon/go-socks5@v0.0.0-20160902184237-e75332964ef5/socks5.go:111 +0x42
github.com/armon/go-socks5.(*Server).ListenAndServe(0x0?, {0x22a7f0f?, 0x22f32ab?}, {0x22f991d?, 0xc0002a22b8?})
	/Users/geekboy/go/pkg/mod/github.com/armon/go-socks5@v0.0.0-20160902184237-e75332964ef5/socks5.go:105 +0x59
github.com/projectdiscovery/proxify.(*Proxy).Run(0xc0001140f0)
	/Users/geekboy/Github/proxify/proxy.go:323 +0x24f
github.com/projectdiscovery/proxify/internal/runner.(*Runner).Run(0xc0002b6fa0)
	/Users/geekboy/Github/proxify/internal/runner/runner.go:122 +0x4fe
main.main()
	/Users/geekboy/Github/proxify/cmd/proxify/proxify.go:33 +0xcf

Write all the logs to single output file when -j option is used.

[INF] Saving proxify traffic to logs/proxify.jsonl

ehsandeep

JSONL output with blank request information, ideally, we should have both request/response information.

cat *jsonl | jq 'select(.url == "")' | jq .request

{
  "header": null,
  "body": "",
  "raw": ""
}
{
  "header": null,
  "body": "",
  "raw": ""
}
{
  "header": null,
  "body": "",
  "raw": ""
}
{
  "header": null,
  "body": "",
  "raw": ""
}
{
  "header": null,
  "body": "",
  "raw": ""
}
{
  "header": null,
  "body": "",
  "raw": ""
}

fix concurent map writes

ehsandeep · 2023-05-17T14:04:07Z

@RamanaReddy0M

proxify $./proxify -j

                       _ ___    
   ___  _______ __ __ (_) _/_ __
  / _ \/ __/ _ \\ \ // / _/ // /
 / .__/_/  \___/_\_\/_/_/ \_, / 
/_/                      /___/

		projectdiscovery.io

[INF] Current proxify version 0.0.9 (latest)
[INF] HTTP Proxy Listening on 127.0.0.1:8888
[INF] Socks5 Proxy Listening on 127.0.0.1:10080
[INF] Saving proxify traffic to logs/proxify.jsonl
fatal error: concurrent map writes

goroutine 3078 [running]:
github.com/projectdiscovery/proxify/pkg/logger.(*Logger).LogResponse(0xc000242c40, 0xc000330120, {{0xc0015c3640, 0x10}, 0x0, 0x1, {0xc000f80c87, 0x13}})
	/Users/geekboy/Github/proxify/pkg/logger/logger.go:212 +0x594
github.com/projectdiscovery/proxify.(*Proxy).ModifyResponse(0xc00013e050, 0xc000330120)
	/Users/geekboy/Github/proxify/proxy.go:158 +0x36b
github.com/projectdiscovery/martian/v3.(*Proxy).handleConnectRequest(0xc00039c600, 0xc00143e1c0?, 0xc00173b200, 0xc0002623c0?, 0xc0002623a0, {0x25e1fa8?, 0xc000e05820})
	/Users/geekboy/go/pkg/mod/github.com/projectdiscovery/martian/v3@v3.0.0-20230412114616-98e3a0a6994a/proxy.go:417 +0x2bd
github.com/projectdiscovery/martian/v3.(*Proxy).handle(0xc00039c600, 0xc00143e140, {0x25e1fa8, 0xc000e05820}, 0xc0002623a0)
	/Users/geekboy/go/pkg/mod/github.com/projectdiscovery/martian/v3@v3.0.0-20230412114616-98e3a0a6994a/proxy.go:608 +0x512
github.com/projectdiscovery/martian/v3.(*Proxy).handleLoop(0xc00039c600, {0x25e1fa8?, 0xc000e05820?})
	/Users/geekboy/go/pkg/mod/github.com/projectdiscovery/martian/v3@v3.0.0-20230412114616-98e3a0a6994a/proxy.go:313 +0x725
created by github.com/projectdiscovery/martian/v3.(*Proxy).Serve
	/Users/geekboy/go/pkg/mod/github.com/projectdiscovery/martian/v3@v3.0.0-20230412114616-98e3a0a6994a/proxy.go:274 +0x478

goroutine 1 [IO wait, 3 minutes]:
internal/poll.runtime_pollWait(0x2d8097c8, 0x72)
	/usr/local/go/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc00039c980?, 0x12d90cb?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc00039c980)
	/usr/local/go/src/internal/poll/fd_unix.go:614 +0x2bd
net.(*netFD).accept(0xc00039c980)
	/usr/local/go/src/net/fd_unix.go:172 +0x35
net.(*TCPListener).accept(0xc000620a80)
	/usr/local/go/src/net/tcpsock_posix.go:148 +0x25
net.(*TCPListener).Accept(0xc000620a80)
	/usr/local/go/src/net/tcpsock.go:297 +0x3d
github.com/armon/go-socks5.(*Server).Serve(0xc0002151d0, {0x25da490, 0xc000620a80})
	/Users/geekboy/go/pkg/mod/github.com/armon/go-socks5@v0.0.0-20160902184237-e75332964ef5/socks5.go:111 +0x42
github.com/armon/go-socks5.(*Server).ListenAndServe(0x0?, {0x22a824f?, 0x22f35f8?}, {0x22f9c6a?, 0xc0001522b8?})
	/Users/geekboy/go/pkg/mod/github.com/armon/go-socks5@v0.0.0-20160902184237-e75332964ef5/socks5.go:105 +0x59
github.com/projectdiscovery/proxify.(*Proxy).Run(0xc00013e050)
	/Users/geekboy/Github/proxify/proxy.go:323 +0x24f
github.com/projectdiscovery/proxify/internal/runner.(*Runner).Run(0xc0002151e0)
	/Users/geekboy/Github/proxify/internal/runner/runner.go:128 +0x570
main.main()
	/Users/geekboy/Github/proxify/cmd/proxify/proxify.go:33 +0xcf

goroutine 16 [sleep]:
time.Sleep(0x3b9aca00)
	/usr/local/go/src/runtime/time.go:195 +0x135
github.com/haxii/fastproxy/servertime.init.0.func1()
	/Users/geekboy/go/pkg/mod/github.com/haxii/fastproxy@v0.5.37/servertime/time.go:12 +0x1e
created by github.com/haxii/fastproxy/servertime.init.0
	/Users/geekboy/go/pkg/mod/github.com/haxii/fastproxy@v0.5.37/servertime/time.go:10 +0x29

Mzack9999

OSX and Linux: LGTM
Windows: the proxify.jsonl file is empty and various txt files are still created.

Mzack9999

lgtm!

fixed

support json output format (#261)

3a02eb1

RamanaReddy0M linked an issue May 15, 2023 that may be closed by this pull request

JSONL output format support #261

Closed

RamanaReddy0M self-assigned this May 15, 2023

RamanaReddy0M marked this pull request as draft May 15, 2023 20:24

RamanaReddy0M added 2 commits May 16, 2023 14:10

fix lint error

2c51570

handle errors

c5776ad

RamanaReddy0M marked this pull request as ready for review May 16, 2023 12:04

RamanaReddy0M requested a review from ShubhamRasal May 16, 2023 12:05

ehsandeep requested changes May 16, 2023

View reviewed changes

ehsandeep previously requested changes May 16, 2023

View reviewed changes

save all logs to proxify.jsonl on jsonl switch

e3dfc38

fix concurent map writes

RamanaReddy0M requested a review from ehsandeep May 17, 2023 12:52

RamanaReddy0M added 2 commits May 17, 2023 21:12

use sync.map instead locks

6ea7370

fix lint error

294a46f

ShubhamRasal requested a review from Mzack9999 May 18, 2023 04:34

Mzack9999 requested changes May 18, 2023

View reviewed changes

RamanaReddy0M requested a review from Mzack9999 May 18, 2023 12:07

RamanaReddy0M and others added 2 commits May 18, 2023 18:21

use userData id as key in sync.Map

9632f5e

removing empty line

e2e2a4b

Mzack9999 approved these changes May 22, 2023

View reviewed changes

ehsandeep approved these changes May 22, 2023

View reviewed changes

ehsandeep merged commit 54d1d63 into dev May 22, 2023
7 checks passed

ehsandeep deleted the issue-261-jsonl-output branch May 22, 2023 07:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

support json output format #294

support json output format #294

RamanaReddy0M commented May 15, 2023 •

edited

ehsandeep left a comment •

edited by RamanaReddy0M

ehsandeep left a comment •

edited by RamanaReddy0M

ehsandeep commented May 17, 2023

Mzack9999 left a comment

Mzack9999 left a comment

support json output format #294

support json output format #294

Conversation

RamanaReddy0M commented May 15, 2023 • edited

ehsandeep left a comment • edited by RamanaReddy0M

Choose a reason for hiding this comment

ehsandeep left a comment • edited by RamanaReddy0M

Choose a reason for hiding this comment

ehsandeep commented May 17, 2023

Mzack9999 left a comment

Choose a reason for hiding this comment

Mzack9999 left a comment

Choose a reason for hiding this comment

RamanaReddy0M commented May 15, 2023 •

edited

ehsandeep left a comment •

edited by RamanaReddy0M

ehsandeep left a comment •

edited by RamanaReddy0M