Add DSL expression based matcher and filter flags #537
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Mzack9999
approved these changes
Jul 27, 2023
| func evalDslExpr(result *Result, dslExpr string) bool { | ||
| resultMap, err := resultToMap(*result) | ||
| if err != nil { | ||
| gologger.Warning().Msgf("Could not map result: %s\n", err) |
There was a problem hiding this comment.
I always think it's better to propagate upstream errors from packages:
func evalDslExpr(result *Result, dslExpr string) (bool, error) { ... }
but since we are already in the output package I think we can keep it this way.
These filtering conditions appears quite powerful also to discriminate which crawling branches abandoning or continue crawling. Shall we track to extend them for this functionality as well in the future? What do you think? (cc @ehsandeep )
There was a problem hiding this comment.
These filtering conditions appears quite powerful also to discriminate which crawling branches abandoning or continue crawling.
@Mzack9999 Yes, let's track it as a new issue. Can you please create an issue for it with minimal information?
Mzack9999
added
the
Type: Enhancement
ehsandeep
requested changes
Jul 27, 2023
There was a problem hiding this comment.
./katana -u https://tesla.com -mdc 'status_code == 200' -headless
__ __
/ /_____ _/ /____ ____ ___ _
/ '_/ _ / __/ _ / _ \/ _ /
/_/\_\\_,_/\__/\_,_/_//_/\_,_/
projectdiscovery.io
[INF] Current katana version v1.0.3-dev (development)
[INF] Started headless crawling for => https://tesla.com
https://www.tesla.com/modules/custom/tesla_banners/js/index.js?v=1.x
https://digitalassets.tesla.com/tesla-contents/image/upload/h_1800,w_2880,c_fit,f_auto,q_auto:best/Homepage-SolarRoof-Desktop-Global
https://www.tesla.com/_flysystem/s3/js/js_ldqeWP3q2sBao83Vz4ByInsqfnYfo1jlDS0TIz5_erw.js
https://www.tesla.com/akam/13/744dae7a
https://www.tesla.com/_flysystem/s3/js/js_aUNhuKdxSPkf8hG7qmD-7aAdSx8hBgwey3Q46cgkgvc.js
https://digitalassets.tesla.com/tesla-contents/image/upload/h_1800,w_2880,c_fit,f_auto,q_auto:best/425_HP_SolarPanels_D
https://digitalassets.tesla.com/tesla-contents/image/upload/h_1800,w_2880,c_fit,f_auto,q_auto:best/Homepage-Model-X-Desktop-LHD
https://digitalassets.tesla.com/tesla-contents/image/upload/h_2400,w_2880,c_fit,f_auto,q_auto:best/Homepage-Model-Y-Global-Desktop
https://www.tesla.com/themes/custom/tesla_frontend/assets/manifest.json
[ERR] Could not evaluate DSL expression: No parameter 'status_code' found.
https://www.tesla.com/_flysystem/s3/css/css__-SMnFsTH-icLbaq2LDPS2K9Zk5bYdTmSnAE4YL-dlQ.css
https://www.tesla.com/_flysystem/s3/css/css_xW-cg_AzHoBA7lMpqERJ5452cKtnXx8j_vFdYDwrYSk.css
https://www.tesla.com/_flysystem/s3/css/css_18rBCG35DW_bd0uLICSc3QZrArz0g_vTHBk_JL7lpso.css
[ERR] Could not evaluate DSL expression: No parameter 'status_code' found.
https://www.tesla.com/_flysystem/s3/js/js_gbNFHvWMw3YIJJyOTXiv51DJoiFkVTiWhP4yGK2Vm9g.js
https://www.tesla.com/_flysystem/s3/js/js_MXgU20utMcyhhXrT_MoiL7oTxtO8Wlx0jf4UwhrNeRg.js
https://digitalassets.tesla.com/tesla-contents/image/upload/h_2560,w_4096,c_fit,f_auto,q_auto:best/Homepage-Model-3-Desktop-LHD
https://www.tesla.com/_flysystem/s3/css/css_F8wWihW99ESm4py_iKtuXvvE0Hse4M2gPjSdO8UfV-g.css
https://digitalassets.tesla.com/tesla-contents/image/upload/f_auto,q_auto:best/Model-S-homepage-desktop
https://digitalassets.tesla.com/tesla-contents/image/upload/f_auto,q_auto:best/Desktop_SolarPanels
https://digitalassets.tesla.com/tesla-contents/image/upload/h_2560,w_4096,c_fit,f_auto,q_auto:best/Homepage-Model-3-Desktop-RHD
[ERR] Could not evaluate DSL expression: No parameter 'status_code' found.
[ERR] Could not evaluate DSL expression: No parameter 'status_code' found.
[ERR] Could not evaluate DSL expression: No parameter 'status_code' found.
[ERR] Could not evaluate DSL expression: No parameter 'status_code' found.
[ERR] Could not evaluate DSL expression: No parameter 'status_code' found.
[ERR] Could not evaluate DSL expression: No parameter 'status_code' found.
[ERR] Could not evaluate DSL expression: No parameter 'status_code' found.
[ERR] Could not evaluate DSL expression: No parameter 'status_code' found.
[ERR] Could not evaluate DSL expression: No parameter 'status_code' found.
[ERR] Could not evaluate DSL expression: No parameter 'status_code' found.Similar to what we did here - projectdiscovery/nuclei#3984
ehsandeep
approved these changes
Jul 31, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed changes:
Add DSL expression-based matcher and filter flags to the katana. Closes #369.
Use cases:
$ katana -u https://tesla.com -mdc 'status_code == 200'