Installing w3af 1.6 on CentOS 6.5

[TomStageDK]

Having been at this for a few days on and off I have come to the conclusion that I can't figure this out without some help.
Following is the system information from the CentOS lsb_release -a:
LSB Version: :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
Distributor ID: CentOS
Description: CentOS release 6.5 (Final)
Release: 6.5
Codename: Final

Installing w3af (This is the installation command found at http://w3af.org/download):
git clone --depth 1 https://github.com/andresriancho/w3af.git

Running w3af_console output:
w3af's requirements are not met, one or more third-party libraries need to be installed.

On Debian systems please install the following operating system packages before running the pip installer:
sudo apt-get install build-essential python-setuptools git python-pip libssl-dev python2.7-dev libsqlite3-dev libxslt1-dev libyaml-dev libxml2-dev

Your python installation needs the following modules to run w3af:
clamd github git.util pybloomfilter esmre phply chardet pdfminer concurrent.futures OpenSSL lxml scapy.config guess_language cluster msgpack ntlm Halberd

After installing any missing operating system packages, use pip to install the remaining modules:
sudo pip install clamd==1.0.1 PyGithub==1.21.0 GitPython==0.3.2.RC1 pybloomfiltermmap==0.3.11 esmre==0.3.1 chardet==2.1.1 pdfminer==20110515 futures==2.1.5 pyOpenSSL==0.13.1 lxml==2.3.2 scapy-real==2.2.0-dev guess-language==0.2 cluster==1.1.1b3 msgpack-python==0.2.4 python-ntlm==1.0.1 halberd==0.2.4
sudo pip install --ignore-installed git+https://github.com/andresriancho/phply.git#egg=phply

A script with these commands has been created for you at /tmp/w3af_dependency_install.sh

Looking in the w3af_console file showed me that it was running python, which is good and all, but w3af needs python 2.7 to run and that is not in any of the repo's I use, so I installed python 2.7.6, using this guide: http://toomuchdata.com/2014/02/16/how-to-install-python-on-centos/
Which gives me a fully functional python 2.7.6 environment on CentOS 6.5, with a python executable name python2.7 in /usr/bin I then changed the w3af_console to use this instead of the default python 2.6.6 by changing the top line to the following:
#!/usr/bin/env python2.7

Running w3af_console once more gave me a list of dependencies that where missing, I then started to find the CentOS equals for the apt-get command that is shown:
sudo apt-get install build-essential python-setuptools git python-pip libssl-dev python2.7-dev libsqlite3-dev libxslt1-dev libyaml-dev libxml2-dev

build-essential dependency:
According to this: http://www.asim.pk/2010/05/28/build-essentials-in-centos/
I should have solved the build-essential requirement with this:
yum groupinstall "Development Tools"
Loaded plugins: downloadonly, fastestmirror, refresh-packagekit, security
Loading mirror speeds from cached hostfile
atomic: mir01.syntis.net
base: mirror.fysik.dtu.dk
epel: ftp.crc.dk
extras: mirror.fysik.dtu.dk
rpmforge: mirror.zetup.net
updates: mirror.fysik.dtu.dk
Setting up Group Process
Checking for new repos for mirrors
Package flex-2.5.35-8.el6.x86_64 already installed and latest version
Package gcc-4.4.7-4.el6.x86_64 already installed and latest version
Package redhat-rpm-config-9.0.3-42.el6.centos.noarch already installed and latest version
Package rpm-build-4.8.0-37.el6.x86_64 already installed and latest version
Package 1:make-3.81-20.el6.x86_64 already installed and latest version
Package patch-2.6-6.el6.x86_64 already installed and latest version
Package 1:pkgconfig-0.23-9.1.el6.x86_64 already installed and latest version
Package gettext-0.17-16.el6.x86_64 already installed and latest version
Package automake-1.11.1-4.el6.noarch already installed and latest version
Package bison-2.4.1-5.el6.x86_64 already installed and latest version
Package libtool-2.2.6-15.5.el6.x86_64 already installed and latest version
Package autoconf-2.63-5.1.el6.noarch already installed and latest version
Package gcc-c++-4.4.7-4.el6.x86_64 already installed and latest version
Package binutils-2.20.51.0.2-5.36.el6.x86_64 already installed and latest version
Package patchutils-0.3.1-3.1.el6.x86_64 already installed and latest version
Package byacc-1.9.20070509-7.el6.x86_64 already installed and latest version
Package indent-2.2.10-7.el6.x86_64 already installed and latest version
Package systemtap-2.3-4.el6_5.x86_64 already installed and latest version
Package diffstat-1.51-2.el6.x86_64 already installed and latest version
Package elfutils-0.152-1.el6.x86_64 already installed and latest version
Package cvs-1.11.23-16.el6.x86_64 already installed and latest version
Package rcs-5.7-37.el6.x86_64 already installed and latest version
Package subversion-1.6.11-10.el6_5.x86_64 already installed and latest version
Package gcc-gfortran-4.4.7-4.el6.x86_64 already installed and latest version
Package 1:doxygen-1.6.1-6.el6.x86_64 already installed and latest version
Package intltool-0.41.0-1.1.el6.noarch already installed and latest version
Package git-1.7.1-3.el6_4.1.x86_64 already installed and latest version
Package ctags-5.8-2.el6.x86_64 already installed and latest version
Package cscope-15.6-6.el6.x86_64 already installed and latest version
Package swig-1.3.40-6.el6.x86_64 already installed and latest version
Warning: Group development does not have any packages.
No packages in any requested group available to install or update

git dependency:
yum list git
Loaded plugins: downloadonly, fastestmirror, refresh-packagekit, security
Loading mirror speeds from cached hostfile
atomic: mir01.syntis.net
base: mirror.fysik.dtu.dk
epel: ftp.crc.dk
extras: mirror.fysik.dtu.dk
rpmforge: mirror.zetup.net
updates: mirror.fysik.dtu.dk
Installed Packages
git.x86_64 1.7.1-3.el6_4.1 @anaconda-CentOS-201311272149.x86_64/6.5

libssl-dev dependency:
yum list openssl
Loaded plugins: downloadonly, fastestmirror, refresh-packagekit, security
Loading mirror speeds from cached hostfile
atomic: mir01.syntis.net
base: mirror.fysik.dtu.dk
epel: ftp.crc.dk
extras: mirror.fysik.dtu.dk
rpmforge: mirror.zetup.net
updates: mirror.fysik.dtu.dk
Installed Packages
openssl.x86_64 1.0.1e-16.el6_5.7 @updates
openssl-devel.x86_64 1.0.1e-16.el6_5.7 @updates
openssl098e.x86_64 0.9.8e-17.el6.centos.2 @anaconda-CentOS-201311272149.x86_64/6.5
pyOpenSSL.x86_64 0.10-2.el6 @anaconda-CentOS-201311272149.x86_64/6.5

libsqlite3-dev dependency:
yum list sqlite*
Loaded plugins: downloadonly, fastestmirror, refresh-packagekit, security
Loading mirror speeds from cached hostfile
atomic: mir01.syntis.net
base: mirror.fysik.dtu.dk
epel: ftp.crc.dk
extras: mirror.fysik.dtu.dk
rpmforge: mirror.zetup.net
updates: mirror.fysik.dtu.dk
Installed Packages
sqlite.x86_64 3.7.9-1.el6.art @atomic
sqlite-devel.x86_64 3.7.9-1.el6.art @atomic

libxslt1-dev dependency:
yum list xslt
Loaded plugins: downloadonly, fastestmirror, refresh-packagekit, security
Loading mirror speeds from cached hostfile
atomic: mir01.syntis.net
base: mirror.fysik.dtu.dk
epel: ftp.crc.dk
extras: mirror.fysik.dtu.dk
rpmforge: mirror.zetup.net
updates: mirror.fysik.dtu.dk
Installed Packages
libxslt.x86_64 1.1.26-2.el6_3.1 @anaconda-CentOS-201311272149.x86_64/6.5
libxslt-devel.x86_64 1.1.26-2.el6_3.1 @anaconda-CentOS-201311272149.x86_64/6.5
libxslt-python.x86_64 1.1.26-2.el6_3.1 @base

libyaml-dev dependency:
yum list yaml
Loaded plugins: downloadonly, fastestmirror, refresh-packagekit, security
Loading mirror speeds from cached hostfile
atomic: mir01.syntis.net
base: mirror.fysik.dtu.dk
epel: ftp.crc.dk
extras: mirror.fysik.dtu.dk
rpmforge: mirror.zetup.net
updates: mirror.fysik.dtu.dk
Installed Packages
PyYAML.x86_64 3.10-7.el6.art @atomic
libyaml.x86_64 0.1.5-1.el6 @epel
libyaml-devel.x86_64 0.1.5-1.el6 @epel

libxml2-dev dependency:
yum list xml2
Loaded plugins: downloadonly, fastestmirror, refresh-packagekit, security
Loading mirror speeds from cached hostfile
atomic: mir01.syntis.net
base: mirror.fysik.dtu.dk
epel: ftp.crc.dk
extras: mirror.fysik.dtu.dk
rpmforge: mirror.zetup.net
updates: mirror.fysik.dtu.dk
Installed Packages
libxml2.x86_64 2.7.6-14.el6 @anaconda-CentOS-201311272149.x86_64/6.5
libxml2-devel.x86_64 2.7.6-14.el6 @anaconda-CentOS-201311272149.x86_64/6.5
libxml2-python.x86_64 2.7.6-14.el6 @anaconda-CentOS-201311272149.x86_64/6.5

According to pip2.7 list (on my system) I have the following Python Modules installed:
pip2.7 list
argparse (1.1)
async (0.6.1)
chardet (2.1.1)
clamd (1.0.1)
cluster (1.1.1b3)
esmre (0.3.1)
futures (2.1.5)
gitdb (0.5.4)
GitPython (0.3.2.RC1)
guess-language (0.2)
halberd (0.2.4)
lxml (2.3.2)
msgpack-python (0.2.4)
nltk (2.0.4)
pdfminer (20110515)
pip (1.5.4)
pybloomfiltermmap (0.3.11)
PyGithub (1.21.0)
pyOpenSSL (0.13.1)
python-ntlm (1.0.1)
python-owasp-zap-v2 (0.0.7)
pythonwhois (2.1.2)
PyYAML (3.11)
requests (2.2.1)
scapy-real (2.2.0-dev)
setuptools (3.4.3)
six (1.4.1)
smmap (0.8.2)
tld (0.6.3)
wsgiref (0.1.2)

According to pip2.7 freeze (on my system) I have the following Python Modules installed:
pip2.7 freeze
GitPython==0.3.2.RC1
PyGithub==1.21.0
PyYAML==3.11
argparse==1.1
async==0.6.1
chardet==2.1.1
clamd==1.0.1
cluster==1.1.1b3
esmre==0.3.1
futures==2.1.5
gitdb==0.5.4
guess-language==0.2
halberd==0.2.4
lxml==2.3.2
msgpack-python==0.2.4
nltk==2.0.4
pdfminer==20110515
pyOpenSSL==0.13.1
pybloomfiltermmap==0.3.11
python-ntlm==1.0.1
python-owasp-zap-v2==0.0.7
pythonwhois==2.1.2
requests==2.2.1
scapy-real==2.2.0-dev
six==1.4.1
smmap==0.8.2
tld==0.6.3
wsgiref==0.1.2

Running w3af_concole now gives the following:
./w3af_console
w3af's requirements are not met, one or more third-party libraries need to be installed.

On Debian systems please install the following operating system packages before running the pip installer:
sudo apt-get install build-essential python-setuptools git python-pip libssl-dev python2.7-dev libsqlite3-dev libxslt1-dev libyaml-dev libxml2-dev

Your python installation needs the following modules to run w3af:
phply

After installing any missing operating system packages, use pip to install the remaining modules:
sudo pip install --ignore-installed git+https://github.com/andresriancho/phply.git#egg=phply

A script with these commands has been created for you at /tmp/w3af_dependency_install.sh

So what dependencies am I missing for this to work?

[andresriancho]

Which gives me a fully functional python 2.7.6 environment on CentOS 6.5, with a python executable name python2.7 in /usr/bin I then changed the w3af_console to use this instead of the default python 2.6.6 by changing the top line to the following:
#!/usr/bin/env python2.7

That sounds good

There might be a bug in w3af's dependency check helper, because centos seems to be supported https://github.com/andresriancho/w3af/blob/master/w3af/core/controllers/dependency_check/platforms/centos.py , and you should get a list of all commands required to install in that platform. Make me a favour and tell me the output of this command:

python -c "import platform; print platform.dist()"

[TomStageDK]

Hi andresriancho
Command:
python2.7 -c "import platform; print platform.dist()"

Responce:
('centos', '6.5', 'Final')

[andresriancho]

Try the new feature branch:

cd w3af
git pull
git checkout feature/fix_2067
./w3af_console

And let me know how that goes.

[TomStageDK]

Now I get the following:
On CentOS systems please install the following operating system packages before running the pip installer:
sudo yum install libsqlite3x-devel

If I do a yum list sqlite there is no package with that name at all, should that be sqlite-devel ?
Which is sqlite-devel.x86_64 3.7.9-1.el6.art from the atomicorp repo.

The above don't exists on a CentOS 5.9 either btw.

[TomStageDK]

Ohhhh and btw thanks for the quick reply and fixes :)

[andresriancho]

Is libsqlite3x-devel the only missing package now? No complaining about any python packages?

[andresriancho]

I'm not a centos user, so I don't really know about the package names. What I can tell you is that if we can find the names of the packages for centos 6.5, then I can modify w3af to check for the right package names.

[TomStageDK]

To answer your first question: Yes
I did a little test on my system where I changed the libsqlite3x-devel to sqlite-devel, and now w3af_console runs, I am about to test if everything works, I will get back to you with the results, and if it is successful I will write up an installation howto for CentOS 6.5 for the website.

[andresriancho]

Awesome! Good to hear you made it 👍

If you've got the time, maybe you should send a pull request for w3af's code, so we check for the right packages in the latest centos! Since there seem to be an older centos, you might have to end up doing some "nasty" is_centos and is_centos_65 functions in https://github.com/andresriancho/w3af/blob/master/w3af/core/controllers/dependency_check/platforms/centos.py

[TomStageDK]

Well I just did a fast scan, with the fast_scan profile, and with output to xml_file, html_file, text_file and export_requests, and it all seems to work, at least there is something in the files I configured, there was only an issue with the profiles not being up to date. Else the scan went along without any problems.

From what I saw today on my CentOS 5.9 and CentOS 6.5 boxes the file should be named sqlite-devel, the file libsqlite3x-devel is NOT present in either of them, so i think it should be enough to just change the file name. I will look into that after easter ( trying to make an install of the 1.6 version on CentOS 5.9 with a custom installation of Python 2.7.6).
I will write an Installation of W3af 1.6 on CentOS 6.5 HowTo and get it to you as fast as I can spare the time :)

[TomStageDK]

Ohhh just tried to start the w3af_gui and that have the same problem as the w3af_console had to start with, for the requirements, it is not detection that it is running on CentOS.

[TomStageDK]

I have made a pull request, and did a change to the centos.py file, but I am not sure I did the pull request correctly. I am new to GitHub so bare with me on this.
I have downloaded the GitHub for windows and it is up and running and I have switched to the feature/fix_2067 branch, so the change should be ready to commit when you allow me to do so.

Have been trying to figure out why the w3af_gui wont start, and why it is not checking the right system requirements, it lists the once for a Debian based system, and even with the graphviz 2.26, gtksourceview2 2.8.2, pygtksourceview 2.8.0, PyGTK 2.16 and GTK 2.20 installed it complaint about the last 2 missing.

[andresriancho]

Commented in the PR

[TomStageDK]

I should now have made the right pull request as described in the Contributing-101 document.

[andresriancho]

All right! After a long wait, all your issues should be solved by the latest commit to develop 👍

Let me explain briefly what happen:

• w3af had a bug (Dependency check default to "Ubuntu" is actually really bad Dependency check default to "Ubuntu" is actually really bad #2703) where if your system wasn't detected/supported by the old dependency_check implementation it defaulted to "Ubuntu". For you it meant that it was trying to verify that your CentOS system had a set of Ubuntu packages installed.
• Fixed Dependency check default to "Ubuntu" is actually really bad #2703 and used it to refactor the whole dependency check thing, which was ugly.
• If you move in the git history to a commit previous to 3c68726 you should be able to start w3af without any issues (if basic dependencies are met, and it won't check any ubuntu stuff)
• 3c68726 helps CentOS 6.5 users with the installation procedure. In other words, if the dependency check detects that you're on centos 6.5 it will show you the commands to install dependencies there; else it will just show you the missing dependencies (generic)