Added two new options: -o/--output and --stdout

[sethsec]

Awesome tool. I've been looking for something like this for a while to integrate with something I am building! I added two new options for you to consider:

1. -o/--output option that allows the user to specify an output directory (overriding the default).

Command:
python photon.py -u http://10.10.10.102:80 -l 2 -t100 -o /pentest/photontest

In this case, all of the output files will be written to /pentest/photontest:

root@htbeu:/pentest/photontest# ls -ltr
total 24
-rw-r--r-- 1 root root    0 Jul 25 11:11 scripts.txt
-rw-r--r-- 1 root root 3260 Jul 25 11:11 robots.txt
-rw-r--r-- 1 root root 3260 Jul 25 11:11 links.txt
-rw-r--r-- 1 root root   17 Jul 25 11:11 intel.txt
-rw-r--r-- 1 root root  437 Jul 25 11:11 fuzzable.txt
-rw-r--r-- 1 root root  146 Jul 25 11:11 files.txt
-rw-r--r-- 1 root root    0 Jul 25 11:11 failed.txt
-rw-r--r-- 1 root root   96 Jul 25 11:11 external.txt
-rw-r--r-- 1 root root    0 Jul 25 11:11 endpoints.txt
-rw-r--r-- 1 root root    0 Jul 25 11:11 custom.txt

2. A --stdout option that allow user to print everything to stdout so they can pipe it into another tool or redirect all output to an output file with an operating system redirector.

Command:
root@htbeu:/opt/dev/Photon# python photon.py -u http://10.10.10.9:80 -l 2 -t100 --stdout

Output:

      ____  __          __
     / __ \/ /_  ____  / /_____  ____
    / /_/ / __ \/ __ \/ __/ __ \/ __ \
   / ____/ / / / /_/ / /_/ /_/ / / / /
  /_/   /_/ /_/\____/\__/\____/_/ /_/

[+] URLs retrieved from robots.txt: 68
[~] Level 1: 69 URLs
[!] Progress: 69/69
[~] Level 2: 9 URLs
[!] Progress: 9/9
[~] Crawling 0 JavaScript files

--------------------------------------------------
[+] URLs: 78
[+] Intel: 1
[+] Files: 1
[+] Endpoints: 0
[+] Fuzzable URLs: 9
[+] Custom strings: 0
[+] JavaScript Files: 0
[+] External References: 3
--------------------------------------------------
[!] Total time taken: 0:32
[!] Average request time: 0.40
[+] Results saved in 10.10.10.9:80 directory

All Results:

http://10.10.10.9:80/themes/*.gif
http://10.10.10.9:80/modules/*.png
http://10.10.10.9:80/INSTALL.mysql.txt
http://10.10.10.9:80/install.php
http://10.10.10.9:80/scripts/
http://10.10.10.9:80/node/add/
http://10.10.10.9:80/?q=admin/
http://10.10.10.9:80/themes/*.png
http://10.10.10.9:80/modules/*.gif
http://10.10.10.9:80
http://10.10.10.9:80/includes/
http://10.10.10.9:80/?q=user/password/
http://10.10.10.9:80/INSTALL.txt
http://10.10.10.9:80/profiles/
http://10.10.10.9:80/themes/bartik/css/ie6.css?on28x3
http://10.10.10.9:80/MAINTAINERS.txt
http://10.10.10.9:80/themes/bartik/css/ie.css?on28x3
http://10.10.10.9:80/modules/*.jpeg
http://10.10.10.9:80/misc/*.gif

The last thing i changed is the way you were wiping the directory each time you ran the tool so that you would get clean output. If you accept the -o option which allows the user to specify the directory, you can't just blindly delete the directory anymore (can't trust user input ;)). So i think i added a cleaner way to just overwrite each file (replacing the w+ with w), that should accomplish the same thing without needing to delete directories.

[s0md3v]

Thanks for fixing the directory wiping problem and adding the --output option but can you describe a scenario where the user would need to redirect the output to another program?
I mean it doesn't contain any useful information that someone would like to supply to some other program.

[sethsec]

I'm writing a tool like vanquish/sniper/etc that runs a whole bunch of tools (gobuster,nikto,whatweb,sqlmap,etc.) and then parses all of that output. it will give you a report of all of the tools output that was run, but it will also parse the output and start recursively running the tools on new directions found by spidering (your tool) or brute force (gobuster).

Gobuster just gives you a list of full urls that it found (no separated output files) and i'd like your tool to do the same so i can then cat them together, remove duplicates, and use that to launch my next wave of tools against any unique directory.

i could parse each of your files and add them each one at a time, but redirecting it to one file is much more in line with what i am doing for all of the other tools:

photon -u url --stdout > 10.10.10.1_80_photon.txt

I guess another way to solve this (for me), is if you add an option that the user can tell the tool to log all of the data to one file:

make sense?

[s0md3v]

Oh I am sorry, I thought you are talking about the output that gets printed on the terminal.
Well I have better idea for this, we can make this option work as follows:

--sdout=storage,files,intel

As you may have guessed, --sdout will take a list of variables separated by commas to send to sdout.

Do you agree with this approach?

[sethsec]

i like it!

[s0md3v]

HI there,
Here are a few things you need to take care of before I can merge it :

1. Please remove code related to stdout completely
2. Can you please check what your code does when no output directory is specified? If there's some code to handle please provide me a permalink to the code because I can't find it :')
3. Change w+ to w in the new file saving code.

Thanks ^_^

[s0md3v]

Great!
Two more things to go!

[sethsec]

So for the output directory, it just works. It is a hack from here: 1adf69b#diff-cafd309176bc7c20679341c01350e33fR396 (lines 396 to 401). If no output directory, it works the way you built it. If there is an output dir, i just change name to = that instead of current directory.

As for the third thing, i'm not sure what you mean about the w+ in the new file saving code.

[s0md3v]

I am talking about this statement of yours:

So i think i added a cleaner way to just overwrite each file (replacing the w+ with w), that should accomplish the same thing without needing to delete directories.

The code responsible for saving the files has been changed so you may want to replace w+ with w there.

[s0md3v]

Do whatever you think is good, just resolve the conflicts :D

[s0md3v]

Any update?

[s0md3v]

$ python photon.py -u "github.com" -l 1 -d 1 -t 10 --regex "\d{10}" --dns
      ____  __          __            
     / __ \/ /_  ____  / /_____  ____ 
    / /_/ / __ \/ __ \/ __/ __ \/ __ \
   / ____/ / / / /_/ / /_/ /_/ / / / /
  /_/   /_/ /_/\____/\__/\____/_/ /_/ 
[+] URLs retrieved from robots.txt: 8
[~] Level 1: 9 URLs
[!] Progress: 9/9
[~] Crawling 1 JavaScript files
[!] Progress: 1/1
[~] Retrieving DNS related data
Traceback (most recent call last):
  File "photon.py", line 446, in <module>
    dnsdumpster(name, colors)
  File "/home/travis/build/s0md3v/Photon/plugins/dnsdumpster.py", line 19, in dnsdumpster
    with open('%s/%s.png' % (domain, domain), 'wb') as f:
IOError: [Errno 2] No such file or directory: 'github.com/github.com.png'

[sethsec]

now that you could potentially be writing to either one of two different places based on the new output param, for the dnsdumpster part, you can just return the image back to photon.py and then it will write it in the right place (either within Photon folder, or within the specified output folder). want me to take a crack at it? or are you working on it?

[s0md3v]

Please revert the change related to dnsdumpster plugin, photon.py isn't supposed to be cluttered.

Do this exactly before file saving related lines

if args.output:
    name = args.output

It's that simple!