system install (as a module) support and setup.py for easier installation

[blshkv]

There are few minor changes would be required. Something like this:

sed -e 's|from discovery|from theHarvester.discovery|' -i theHarvester.py || die "sed failed"
sed -e 's|from lib|from theHarvester.lib|' -i theHarvester.py || die "sed failed"
sed -e 's|from lib|from theHarvester.lib|' -i lib/htmlExport.py || die "sed failed"
for i in discovery/*.py; do
     sed -e 's|import myparser|from theHarvester import myparser|' -i $i || die "sed for $i failed"
done
touch __init__.py

[blshkv]

I noted that there is a new sqlite database in use. The DB should be generated in homedir ~/.theHarverster
from scratch without any need in a template. The wordlist can be installed into a shared directory.

[NotoriousRebel]

@blshkv would you want to create an install.sh for this? There is currently a dockerfile and you can also git clone it. Sorry for the late response!

[blshkv]

no, that would be a wrong approach. It has to be setup.py which should list the required dependencies and define system location of all files. The tool should also support system-wide installation, i.e. installed under root into /usr/share or /usr/lib{32,64} and a regular user should be able to run it. That would also require to move all write operations (database, outputs) from a current directory into
a current user ~/.theHarvester or similar.

[NotoriousRebel]

@blshkv would you want to work on this and submit a pull request?

[blshkv]

Sorry, I don't have bandwidth for development of other tools. Just a simple bug reporting (like this issue) and maintaining hundreds of packages takes a fair amount of time which I could afford.

[L1ghtn1ng]

Hi All, I have updated the documentation for the dependencies but it needs to be merge and when you run the tool for the first time it creates the sqlite DB

[L1ghtn1ng]

@blshkv My changes have been merged can you take a look at the README.md please?

[blshkv]

Not sure what to look for. Does the tool support system install using setup.py?
No.

Will it only from the current folder? Yes.
So nothing changed

[L1ghtn1ng]

The dependencies section and will look to getting the Kali patch upstreamed which should fix this

…

________________________________ From: Anton Bolshakov <notifications@github.com> Sent: Tuesday, January 1, 2019 5:36:08 AM To: laramies/theHarvester Cc: J.Townsend; Comment Subject: Re: [laramies/theHarvester] system install support and setup.py for easier installation (#52) Not sure what to look for. Does the tool support system install using setup.py? No. Will it only from the current folder? Yes. So nothing changed — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#52 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ADddQpAcTC_n3OLlZ54aELf-iCgHHfmOks5u-vPIgaJpZM4MmG8z>.

[blshkv]

I have tried it, still long way to go.

setup.py is a python file, which usually tells you that the module/package you are about to install has been packaged and distributed with Distutils, which is the standard for distributing Python Modules.

This allows you to easily install Python packages. Often it's enough to write:

$ python setup.py install

and the module will install itself.

https://docs.python.org/3/installing/index.html#installing-index

[NotoriousRebel]

@blshkv what do you think of the PR for setup.py?

[blshkv]

I have tried to convert the harvester into a proper python module but this is a major re-design and I'm unable to test all functions. As I said, I have no time to rewrite it, however I will be happy to test your progress. I can also help with minor fixes, once the direction of the development is clear.

[leebaird]

Please review the new setup.py file that was added. Would like to close out this ticket.

[blshkv]

Tested and failed. Have you tested at all?
The current setup.py doesn't install wordlists folder and theHarvester.py.
I copied the binary into /usr/bin manually but the tool failed to start:

$ theHarvester.py 
Traceback (most recent call last):
  File "/usr/bin/theHarvester.py", line 3, in <module>
    from discovery import *
  File "/usr/lib64/python3.6/site-packages/discovery/shodansearch.py", line 3, in <module>
    from shodan import exception
ModuleNotFoundError: No module named 'shodan'

You need to convert it into python module, redirect all outputs into user's dir etc . Still long way to go

[blshkv]

Regarding the wordlists, please use the following example:
derv82/wifite2#102
They used data_files function for this purpose

[NotoriousRebel]

@blshkv please try now I tested it after the most recent commit and it seems to be working :)

[blshkv]

worldlist files are installed into an unusual location: /usr/worldlist now, all other issues are not fixed

[blshkv]

FYI, you can debug it with the following command:
python3 setup.py install --user and inspect ~/.local/lib64/python3.6

[blshkv]

#229
I have started to redesign the tool here (spent like 2 hours on it), let's see if you agree with it.

[blshkv]

https://github.com/vulnersCom/getsploit/pull/14/files

[NotoriousRebel]

@blshkv if you are satifised with the changes that have happened feel free to close :)

[blshkv]

@NotoriousRebel thank you for accepting the patch. That would be the first stage, I can install the tool as a system package now (using root account). The second stage is to redirect all output to a userdir, (something like https://github.com/vulnersCom/getsploit/pull/14/files which I mentioned before) and redirect all output to ~/.theHarvester so a regular user would be able to run it without touching system files

[L1ghtn1ng]

@blshkv Would you agree that this issue could be closed now? As I think we now for fill the subject of this thread and open a new thread to do with the other bits and pieces that you have mentioned.

[blshkv]

Ok, sure. Thanks for accepting my patch once again