requirements.txt problem

[Sydwicked]

root@kali:~/xsscrapy# pip install -r requirements.txt
Collecting Scrapy==1.1.0rc3 (from -r requirements.txt (line 1))
Using cached https://files.pythonhosted.org/packages/c4/33/a87d324a3c25b6e6e8018b9161987e185910bd6e611ebb75ce169a7f1312/Scrapy-1.1.0rc3-py2.py3-none-any.whl
Collecting pybloom==1.1 (from -r requirements.txt (line 2))
Using cached https://files.pythonhosted.org/packages/1a/82/a1ad015bdc19bd7e10aa97070329b84b5e01c0c6b5de88df664a98413eed/pybloom-1.1.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "", line 1, in
File "/tmp/pip-install-kysj5guf/pybloom/setup.py", line 2, in
from ez_setup import use_setuptools
File "/tmp/pip-install-kysj5guf/pybloom/ez_setup.py", line 98
except pkg_resources.VersionConflict, e:
^
SyntaxError: invalid syntax

----------------------------------------

Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-kysj5guf/pybloom/

[decidedlygray]

Does this fix the issue?

pip install --upgrade setuptools
pip install -r requirements.txt

[Sydwicked]

Does this fix the issue?

pip install --upgrade setuptools
pip install -r requirements.txt

Nope

Root@kali:~# pip install --upgrade setuptools
Collecting setuptools
Downloading https://files.pythonhosted.org/packages/82/a1/ba6fb41367b375f5cb653d1317d8ca263c636cff6566e2da1b0da716069d/setuptools-40.5.0-py2.py3-none-any.whl (569kB)
100% |████████████████████████████████| 573kB 1.1MB/s
Installing collected packages: setuptools
Found existing installation: setuptools 40.4.3
Uninstalling setuptools-40.4.3:
Successfully uninstalled setuptools-40.4.3
Successfully installed setuptools-40.5.0

root@kali:/xsscrapy# ls
README.md requirements.txt scrapy.cfg tests xsscrapy xsscrapy.py
root@kali:/xsscrapy# pip install -r requirements.txt
Collecting Scrapy==1.1.0rc3 (from -r requirements.txt (line 1))
Using cached https://files.pythonhosted.org/packages/c4/33/a87d324a3c25b6e6e8018b9161987e185910bd6e611ebb75ce169a7f1312/Scrapy-1.1.0rc3-py2.py3-none-any.whl
Collecting pybloom==1.1 (from -r requirements.txt (line 2))
Using cached https://files.pythonhosted.org/packages/1a/82/a1ad015bdc19bd7e10aa97070329b84b5e01c0c6b5de88df664a98413eed/pybloom-1.1.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "", line 1, in
File "/tmp/pip-install-e5_clvfr/pybloom/setup.py", line 2, in
from ez_setup import use_setuptools
File "/tmp/pip-install-e5_clvfr/pybloom/ez_setup.py", line 98
except pkg_resources.VersionConflict, e:
^
SyntaxError: invalid syntax

----------------------------------------

Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-e5_clvfr/pybloom/
root@kali:~/xsscrapy#

[LennonCMJ]

It working for me
pip install --upgrade setuptools
pip install -r requirements.txt

[rcha0s]

I am seeing the same error. Can this be looked at please?

[osamahamad]

I had this too and

pip install --upgrade setuptools
pip install -r requirements.txt

does not fix the problem...

/xsscrapy# pip install -r requirements.txt
Collecting Scrapy==1.1.0rc3
  Using cached Scrapy-1.1.0rc3-py2.py3-none-any.whl (292 kB)
Collecting pybloom==1.1
  Using cached pybloom-1.1.tar.gz (10 kB)
    ERROR: Command errored out with exit status 1:
     command: /usr/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-_k89_9hy/pybloom/setup.py'"'"'; __file__='"'"'/tmp/pip-install-_k89_9hy/pybloom/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-_k89_9hy/pybloom/pip-egg-info
         cwd: /tmp/pip-install-_k89_9hy/pybloom/
    Complete output (8 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-_k89_9hy/pybloom/setup.py", line 2, in <module>
        from ez_setup import use_setuptools
      File "/tmp/pip-install-_k89_9hy/pybloom/ez_setup.py", line 98
        except pkg_resources.VersionConflict, e:
                                            ^
    SyntaxError: invalid syntax
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

[decidedlygray]

@osamahamad it looks like you’re trying it with Python3 - try using Python2

[osamahamad]

@decidedlygray I tried with python2 & python3 and the same error occured.

[decidedlygray]

@osamahamad can you post the full stack trace for python2 like you did for python3 above?

[imchiragprajapati]

@decidedlygray Sir,
As you've said to use Python2 !
I have tried all my possibility then too it's not working even my friend who i suggested to try to install this tool, even He is not be able to install xsscrapy !

[decidedlygray]

Are you able to post the /tmp/pip-record-8IxLs5/install-record.txt?

[decidedlygray]

Actually, it looks like the x86_64-linux-gnu-gcc command is failing. You may have some unmet linux dependencies. This can probably be fixed with:

sudo apt-get install -y python-dev build-essential libssl-dev libffi-dev libxml2-dev libxslt1-dev zlib1g-dev python-pip

[imchiragprajapati]

@decidedlygray Sir,
I have tried to the above command given and it is Showing me an error (attachment)

[shubham49m]

i am facing requirements.txt issue in linux

[decidedlygray]

@ImPRINCE99 I looked into this a little bit. It looks like they've removed that package.

Try this, install the other dependencies:

sudo apt update
sudo apt-get install -y python-dev build-essential libssl-dev libffi-dev libxml2-dev libxslt1-dev zlib1g-dev

Then install pip2 manually:

curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python2 get-pip.py

@shubham49m I have responded to this question in the ticket you opened: #56

[imchiragprajapati]

Sir, As you've said I had done step by step but the error's are still coming, Please Help !

[decidedlygray]

The problem here is pip is calling python3 (I still see /usr/bin/python3 -c in that error message).

I just tested the steps below on a new Kali setup (2020.4), and it appears to work. It should work for any new version of kali that people are seeing this problem with. Follow along at your own risk. I'll try to upgrade xsscrapy to python3 and put in a pull request soon.

5 Steps to Get xsscrapy Running on New Kali Installations

Step 1 - Install Pyenv

Since we don't want to break the python3 stuff in Kali, let's install python2 using Pyenv so we can switch back and forth.

To install it, just run the following commands:

sudo apt update && sudo apt upgrade -y
sudo apt-get install -y build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev xz-utils tk-dev libffi-dev liblzma-dev python3-openssl git
curl https://pyenv.run | bash

Step 2 - Make ZSH aware of our changes and reload our shell

Now let's make sure ZSH uses pyenv by running the following commands:

echo '' >> ~/.zshrc
echo '# pyenv stuff'
echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.zshrc
echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.zshrc
echo -e 'if command -v pyenv 1>/dev/null 2>&1; then\n  eval "$(pyenv init -)"\nfi' >> ~/.zshrc
exec $SHELL

Step 3 - Change our python to point to python2

With pyenv install, now let's point python to python2 by running the following commands:

pyenv install 2.7.18
pyenv global 2.7.18
export PYTHONPATH=/home/kali/.pyenv/versions/2.7.18/lib/python2.7/site-packages/:$PYTHONPATH

Step 4 - Install the project requirements

You have probably done this first part, but we'll of course need to clone xsscrapy so we have it:

cd /opt
sudo -E git clone https://github.com/DanMcInerney/xsscrapy
cd xsscrapy

And now we install the requirements:

pip install -r requirements.txt

Step 5 - Run xsscrapy!

Now the fun part (hopefully), run xsscrapy!

Close your terminal window (yep, this is required).

Open a new one and you should be able to run xsscrapy:

cd /opt/xsscrapy
python xsscrapy.py -h

Additional Notes about Your New Setup

Don't run this now, but if you ever need to switch back to Python3 as your default version of python, just run the following command:

pyenv global system

btw, I pulled all this information from the following resources. Blame may go to me, but honor and glory go to the authors of these resources:

https://www.kali.org/docs/general-use/python3-transition/
https://www.kali.org/docs/general-use/using-eol-python-versions/

@ImPRINCE99 Let me know if that worked. It looks like you might be using a slightly older version of Kali, but still released within the past year.

[imchiragprajapati]

I used the command that you mentioned and i still face the mitmproxy error.
i have attached screenshot for reference, please look into it
Thanks :)

[decidedlygray]

It looks like you’re using an older version of Kali. mitmproxy is a separate project, it looks like it might not be compatible with upgraded packages. It might be time to upgrade Kali (sudo apt update && sudo apt upgrade -y && sudo apt update && sudo apt dist-upgrade) then try it. Warning though: dist-upgrade broke my Kali and I had to rebuild. So if your Kali is critical for getting your work done or anything you might want to set up a separate VM with a new Kali and move over to that

[Broly157]

hey @decidedlygray
Yes, I tried these steps but still not working, Maybe bcoz i'm using kali 2020.4v? Coz when is tried it on my friends VPS Ubuntu version 2018 smthg their it worked just fine.
I'm getting this Error Attached Pic

Thanks

[decidedlygray]

@Broly157 Some step was skipped or didn’t work properly (maybe step 3) because I can see from your error message that you’re using python3 and not 2

[Broly157]

@Broly157 Some step was skipped or didn’t work properly (maybe step 3) because I can see from your error message that you’re using python3 and not 2

Yes, I'm using python3, I have to use python2?
Thanx

[decidedlygray]

Yes

[decidedlygray]

The 5 steps provided are how to get python2 up and running.

An alternative would be to use the python3 work in progress version here, but it sounds like it is somewhat untested - https://github.com/L1NT/xsscrapy

[ptrk01]

Facing the same problems. Considering to migrate XSScrapy to Python3. Has anyone tried or done this?

[decidedlygray]

Indeed. See my comment directly above yours - it is a test version converted to python3