Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

improved logging #278

Merged
merged 1 commit into from
Jan 23, 2023
Merged

improved logging #278

merged 1 commit into from
Jan 23, 2023

Conversation

tarunKoyalwar
Copy link
Member

@tarunKoyalwar tarunKoyalwar commented Jan 20, 2023

Proposed Changes

  • Now log contains host/url in crawl warning/errors
  • Improved Error logging using utils/errors
  • removed deprecated github.com/pkg/errors
  • Added debug mode support with env variable DEBUG=1
    • Prints stacktrace of error if DEBUG variable is not empty

closes #168

@tarunKoyalwar
Copy link
Member Author

$ ./katana -u https://feedly.com -d 5 -jc -proxy http://127.0.0.1:8080 -v

   __        __                
  / /_____ _/ /____ ____  ___ _
 /  '_/ _  / __/ _  / _ \/ _  /
/_/\_\\_,_/\__/\_,_/_//_/\_,_/ v0.0.3							 

		projectdiscovery.io

[WRN] Could not request seed URL https://feedly.com: remote error: tls: user canceled
$ ./katana -u https://feedly.com/path\?with\=param -d 5 -jc -proxy http://127.0.0.1:8080 -v

   __        __                
  / /_____ _/ /____ ____  ___ _
 /  '_/ _  / __/ _  / _ \/ _  /
/_/\_\\_,_/\__/\_,_/_//_/\_,_/ v0.0.3							 

		projectdiscovery.io

[header] https://feedly.com/i/my
[WRN] Could not request seed URL https://feedly.com/i/my: remote error: tls: user canceled
[WRN] Could not request seed URL https://feedly.com/path?with=param: remote error: tls: user canceled

@tarunKoyalwar tarunKoyalwar self-assigned this Jan 20, 2023
@tarunKoyalwar
Copy link
Member Author

tarunKoyalwar commented Jan 20, 2023

Descriptive Errors with utils/errors

$ ./katana -u https://nosdfsfs -v -hl 

   __        __                
  / /_____ _/ /____ ____  ___ _
 /  '_/ _  / __/ _  / _ \/ _  /
/_/\_\\_,_/\__/\_,_/_//_/\_,_/ v0.0.3							 

		projectdiscovery.io

[WRN] Could not request seed URL https://nosdfsfs: [hybrid:RUNTIME] navigation failed: net::ERR_NAME_NOT_RESOLVED <- could not navigate target

Using DEBUG to pinpoint errors (for debugging purposes)

$ DEBUG=1 ./katana -u https://nosdfsfs -v -hl

   __        __                
  / /_____ _/ /____ ____  ___ _
 /  '_/ _  / __/ _  / _ \/ _  /
/_/\_\\_,_/\__/\_,_/_//_/\_,_/ v0.0.3							 

		projectdiscovery.io

[WRN] Could not request seed URL https://nosdfsfs: [hybrid:RUNTIME] navigation failed: net::ERR_NAME_NOT_RESOLVED <- could not navigate target
Stacktrace:
goroutine 63 [running]:
runtime/debug.Stack()
	/opt/homebrew/Cellar/go/1.19.4/libexec/src/runtime/debug/stack.go:24 +0x64
github.com/projectdiscovery/utils/errors.(*enrichedError).captureStack(...)
	/Users/tarun/go/pkg/mod/github.com/projectdiscovery/utils@v0.0.4/errors/enriched.go:117
github.com/projectdiscovery/utils/errors.(*enrichedError).Error(0x14000554140)
	/Users/tarun/go/pkg/mod/github.com/projectdiscovery/utils@v0.0.4/errors/enriched.go:56 +0x198
fmt.(*pp).handleMethods(0x140002668f0, 0xd3a40?)
	/opt/homebrew/Cellar/go/1.19.4/libexec/src/fmt/print.go:651 +0x18c
fmt.(*pp).printArg(0x140002668f0, {0x1012ea960?, 0x14000554140}, 0x73)
	/opt/homebrew/Cellar/go/1.19.4/libexec/src/fmt/print.go:740 +0x6c0
fmt.(*pp).doPrintf(0x140002668f0, {0x100d9b74f, 0x22}, {0x14001795f50?, 0x2, 0x2})
	/opt/homebrew/Cellar/go/1.19.4/libexec/src/fmt/print.go:1057 +0x21c
fmt.Sprintf({0x100d9b74f, 0x22}, {0x14001795f50, 0x2, 0x2})
	/opt/homebrew/Cellar/go/1.19.4/libexec/src/fmt/print.go:219 +0x4c
github.com/projectdiscovery/gologger.(*Event).Msgf(0x140004ea240, {0x100d9b74f?, 0x140000d3a40?}, {0x14001795f50?, 0x0?, 0x0?})
	/Users/tarun/go/pkg/mod/github.com/projectdiscovery/gologger@v1.1.7/gologger.go:139 +0x3c
github.com/projectdiscovery/katana/pkg/engine/hybrid.(*Crawler).Crawl.func3()
	/Users/tarun/Codebase/katana/pkg/engine/hybrid/hybrid.go:228 +0x288
created by github.com/projectdiscovery/katana/pkg/engine/hybrid.(*Crawler).Crawl
	/Users/tarun/Codebase/katana/pkg/engine/hybrid/hybrid.go:216 +0xaa0

@tarunKoyalwar tarunKoyalwar linked an issue Jan 20, 2023 that may be closed by this pull request
@ehsandeep ehsandeep merged commit e62870f into dev Jan 23, 2023
@ehsandeep ehsandeep deleted the issue-168-error-description branch January 23, 2023 09:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Logging improvement to include requested host/URI
3 participants