[feature] Add feature to remove timestamp from results

[Tedixx]

Is your feature request related to a problem? Please describe.
Due to the timestamp, it is hard to sort unique vulnerabilities, e.g. with anew.

Describe the solution you'd like
Add flag -no-timestamp flag to remove the timestamp from output

[geeknik]

Just in case it takes a few cycles for them to pick up this request, this is an easy post-process fix:

cat nuclei-output | sed -r 's/\[[0-9]{1,10}:[0-9]{2}:[0-9]{2}\]//' | anew

Good luck out there! \m/

[Tedixx]

Just in case it takes a few cycles for them to pick up this request, this is an easy post-process fix:

cat nuclei-output | sed -r 's/\[[0-9]{1,10}:[0-9]{2}:[0-9]{2}\]//' | anew

Good luck out there! \m/

Thanks for your comment! However, while using this command, the output still remains with the timestamp (and causes anew to fail)

[geeknik]

Thanks for your comment! However, while using this command, the output still remains with the timestamp (and causes anew to fail)

Yeah, that's my bad, I failed to take into account the date in the timestamp, try this instead:

sed -r 's/\[[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{1,10}:[0-9]{2}:[0-9]{2}\]//'

[forgedhallpass]

Hello @Tedixx,

This can be achieved easily in multiple ways. One would be what @geeknik showed, but the easiest is probably to use:
cat nuclei-putput | sort -k 3 # sort based on the 3rd column (default separator is the space character
cat nuclei-ouput | cut -f 3- -d' ' | sort # only shows columns starting from the 3rd column until the end, where the delimiter is a space character
cat nuclei-ouput | awk '{for(i=3;i<=NF;i++)printf $i""FS; print ""}' | sort # awk removes the first 2 columns, then sorts the result

Example outputs:

$ cat nucleiOutput-14.txt
[2021-06-16 19:16:22] [wordpress-directory-listing] [http] [info] http://localhost/wp-includes/
[2021-06-16 19:16:26] [tech-detect:wordpress] [http] [info] http://localhost
[2021-06-16 19:16:26] [tech-detect:apache] [http] [info] http://localhost
[2021-06-16 19:16:26] [tech-detect:php] [http] [info] http://localhost
[2021-06-16 19:16:30] [wp-license-file] [http] [info] http://localhost/license.txt

$ cat nucleiOutput-14.txt | sort -k3
[2021-06-16 19:16:26] [tech-detect:apache] [http] [info] http://localhost
[2021-06-16 19:16:26] [tech-detect:php] [http] [info] http://localhost
[2021-06-16 19:16:26] [tech-detect:wordpress] [http] [info] http://localhost
[2021-06-16 19:16:22] [wordpress-directory-listing] [http] [info] http://localhost/wp-includes/
[2021-06-16 19:16:30] [wp-license-file] [http] [info] http://localhost/license.txt

$ cat nucleiOutput-14.txt | cut -f 3- -d' ' | sort
[tech-detect:apache] [http] [info] http://localhost
[tech-detect:php] [http] [info] http://localhost
[tech-detect:wordpress] [http] [info] http://localhost
[wordpress-directory-listing] [http] [info] http://localhost/wp-includes/
[wp-license-file] [http] [info] http://localhost/license.txt
admin@ip-172-31-14-91:/var/lib/jenkins/workspace/test-maste

In my opinion it does not worth adding a new flag to achieve this.

[Tedixx]

Thanks for your feedback @geeknik and @forgedhallpass . Your commands are helpful and get most of the job done. However, I still feel like the -no-timestamp command is needed due to the following reasons;

1. Ease of use; nuclei is a flexible tool which allows to search for vulnerabilities quickly and easy. The -no-timestamp command will help users to automate their workflow (e.g. /w anew), without the hassle of using regex/awk/cut.
2. Minimizing errors: The output of nuclei templates changes continiously, which could cause regex/awl/cut to fail. For example; I've used your ack command to parse a large nuclei result list. I've encountered this error;

awk: cmd. line:1: (FILENAME=- FNR=81) fatal: not enough arguments to satisfy format string
        `https://sub.example.com/%c0 '
                                      ^ ran out for this one

This caused that only half of my results were being shown. By introducing a -no-timestamp flag, this error could have been prevented.

In my opinion, those two reasons show it is worth adding a new flag.

[forgedhallpass]

As mentioned in my previous comment, the easiest way is just to use sort -k3 or use cut -f 3- -d' '. Adding CLI flags for every little thing would very rapidly become unsustainable. A better option in this case would rather be to let a logger pattern control the output from within the configuration file, so then you could define your output structure the way you want. For example something like: [%yyyy-%MM-%dd %HH:%mm:%ss] [%tid] [%proto] %url

p.s. I'd like to take a look at an anonymized outputfile of yours to figure out what the actual problem with my awk comment was...just for the sake of curiousity.

[ehsandeep]

It looks like the timestamp information was intended to add into JSON output only - #468 as such we can also consider to remove this information from CLI output to make it automation friendly.

[geeknik]

It looks like the timestamp information was intended to add into JSON output only - #468 as such we can also consider to remove this information from CLI output to make it automation friendly.

I hope that means a -no-time or -timestamp flag as we specifically requested the time stamp for CLI output in addition to the JSON output. 👍🏻

[Tedixx]

In my opinion, every command would do, as long as nuclei prints output without timestamp (and no additional command is needed). A config approach like [%yyyy-%MM-%dd %HH:%mm:%ss] [%tid] [%proto] %url seems like a fair solution, which also allows further modification for automation. But also I'm quite font of -no-time or -timestamp due the ease of use.

@forgedhallpass I will send you the output file later today/tomorrow.

[ehsandeep]

@Tedixx This flag has been added to disable printing time-stamp information to CLI here b7e3eec, we are also considering having a single CLI flag in the future that can define/control all the information gets printed to CLI, so in that way, users can have complete control over all the fields printed to CLI.

[Tedixx]

Thanks a lot!