Add katana as lib

[ShubhamRasal]

This PR adds

• callback function for result found
• katana as library example
• integration tests for library code

package main

import (
	"github.com/projectdiscovery/gologger"
	"github.com/projectdiscovery/katana/pkg/engine/standard"
	"github.com/projectdiscovery/katana/pkg/output"
	"github.com/projectdiscovery/katana/pkg/types"
)

func main() {
	options := &types.Options{
		MaxDepth:     1,               // Maximum depth to crawl
		FieldScope:   "rdn",           // Crawling Scope Field
		BodyReadSize: 2 * 1024 * 1024, // Maximum response size to read
		RateLimit:    150,             // Maximum requests to send per second
		OnResult: func(result output.Result) { // Callback function to execute for result
			gologger.Info().Msg(result.URL)
		},
	}
	crawlerOptions, err := types.NewCrawlerOptions(options)
	if err != nil {
		gologger.Fatal().Msg(err.Error())
	}
	crawler, err := standard.New(crawlerOptions)
	if err != nil {
		gologger.Fatal().Msg(err.Error())
	}
	var input = "https://tesla.com"
	err = crawler.Crawl(input)
	if err != nil {
		gologger.Warning().Msgf("Could not crawl %s: %s", input, err.Error())
	}
}

[Mzack9999]

While the implementation is correct - It puts in evidence that to properly use katana as a library, we would need the following:

• OnResult callback
• Integration tests to verify the introduced example

I'd suggest covering at least the first point and tracking the second into another issue. What do you think?

[ShubhamRasal]

While the implementation is correct - It puts in evidence that to properly use katana as a library, we would need the following:

• OnResult callback
• Integration tests to verify the introduced example

I'd suggest covering at least the first point and tracking the second into another issue. What do you think?

Yeah.. I was thinking the same.. would add the callback as well
Have added the integration test in the same PR. as they are dependent.

[Mzack9999]

lgtm - suggesting a target change (scanme.sh). Maybe we should expose a more complex webapp for testing (cc @ehsandeep )

[ehsandeep]

@Mzack9999 as of now replaced tesla.com with public-firing-range.appspot.com as scanme need to be updated to include URLs/endpoint for testing.