Display Source URL for Extracted Fields when Using custom.yaml in Katana

[kayra-s4e]

katana version:

v1.1.2 (latest)

Current Behavior:

When using a custom.yaml file and extracting a specific field with Katana, the output does not show the source URL from which the field was found.

Expected Behavior:

When using a field from custom.yaml, the output should include the source URL along with the extracted field.

Example Command:

katana -u https://redacted.com/ -flc custom.yaml -f tckno

Expected Output:

39478627938 ---> https://redacted.com/index.php
98575212324 ---> https://redacted.com/index.php
12345678950 ---> https://redacted.com/contact.php
44455354000 ---> https://redacted.com/index.php
23456789012 ---> https://redacted.com/about.php
10012345000 ---> https://redacted.com/index.php

Steps To Reproduce:

1. Create a custom.yaml file with a field definition.
2. Run Katana with the -flc and -f options.
3. Observe that the extracted field is shown, but the source URL is missing from the output.

Anything else:

Adding the source URL to the output when using custom fields would improve traceability and usability. Implementing a flag to enable this functionality when using the -f parameter would also be beneficial.

Thank you!

[ehsandeep]

@kayra-s4e source information is part of jsonl output, did you see it missing in jsonl or CLI?

[kayra-s4e]

Yes, I have also tried using the -j parameter. When I use -j to output in JSON format, it writes all the crawled endpoints to the output file. However, in this scenario, even though 7 TCK numbers were found, the JSON output contains 15 lines. How can I determine which TCK number belongs to which source?

[dogancanbakir]

@dwisiswant0 I think we can export custom fields in JSON format(as part of JSON output) to enable this mapping.

[kayra-s4e]

any update??

[kayra-s4e]

hi, is there any progress?