TWARC Utilities

[shamreeza]

Hi while doing the TWARC tutorial, I have encounter following errors. Would appreciate any advice how to resolve. Thanks
1.
Shamreezas-MacBook-Air:#COVID19Vic shamreezariaz$ utils/d3_network.py tweets.jsonl tweet_graph.html
-bash: utils/d3_network.py: No such file or directory
2.
Shamreezas-MacBook-Air:#COVID19Vic shamreezariaz$ utils/wall.py tweets.jsonl > tweets.html
Traceback (most recent call last):
File "utils/wall.py", line 17, in
import requests
ImportError: No module named requests

[igorbrigadir]

Which tutorial are you following? Do you have a link?

for 1:

I assume you intend to run https://github.com/DocNow/twarc/blob/main/utils/network.py ? In that case, it may be that the documentation isn't correct, the file name should be network.py not d3_network.py

for 2:

The No module named requests is nearly always a symptom of not using the correct environment if you installed twarc correctly. While the scripts are executable, it's better to explicitly run them with a python command, and make sure that it's the right python version. This depends on how you installed twarc in the first place, but if this doesn't work:

python utils/wall.py tweets.jsonl > tweets.html

try running this

python3 utils/wall.py tweets.jsonl > tweets.html

in the terminal instead.

[edsu]

Thanks for the question @shamreeza and for responding @igorbrigadir! I think this issue really speaks to how twarc utilities would be better aligned as installable packages that could be invoked via a subcommand with twarc. We've been discussing this a bit for twarc2. If they were installable the dependencies and documentation could be managed better. The current approach to the utils directory as a kitchen sink was good for idea generation but has proven difficult to manage and use over time.

[shamreeza]

Thanks for your response. I tried many times but problem is still there
utils/network.py tweets.jsonl network.html
Traceback (most recent call last):
File "utils/network.py", line 26, in
import networkx
ImportError: No module named networkx

[shamreeza]

Shamreezas-MacBook-Air:Ms Higgins shamreezariaz$ pip3 install networks
Collecting networks
Downloading networks-0.3.7.tar.gz (8.2 kB)
Collecting jsonpickle
Downloading jsonpickle-2.0.0-py2.py3-none-any.whl (37 kB)
Requirement already satisfied: numpy in /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages (from networks) (1.20.1)
Requirement already satisfied: scipy in /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages (from networks) (1.6.1)
Requirement already satisfied: matplotlib in /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages (from networks) (3.3.4)
Requirement already satisfied: pillow>=6.2.0 in /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages (from matplotlib->networks) (8.1.0)
Requirement already satisfied: python-dateutil>=2.1 in /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages (from matplotlib->networks) (2.8.1)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.3 in /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages (from matplotlib->networks) (2.4.7)
Requirement already satisfied: kiwisolver>=1.0.1 in /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages (from matplotlib->networks) (1.3.1)
Requirement already satisfied: cycler>=0.10 in /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages (from matplotlib->networks) (0.10.0)
Requirement already satisfied: six in /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages (from cycler>=0.10->matplotlib->networks) (1.15.0)
Using legacy 'setup.py install' for networks, since package 'wheel' is not installed.
Installing collected packages: jsonpickle, networks
Running setup.py install for networks ... done
Successfully installed jsonpickle-2.0.0 networks-0.3.7
Shamreezas-MacBook-Air:Ms Higgins shamreezariaz$ utils/network.py tweets.jsonl tweet_graph.html
Traceback (most recent call last):
File "utils/network.py", line 26, in
import networkx
ImportError: No module named networkx
Shamreezas-MacBook-Air:Ms Higgins shamreezariaz$

Please suggest if there is any error. I am not from this field and trying TWARC by using tutorial. https://scholarslab.github.io/learn-twarc/07-utilities-mac

[igorbrigadir]

Please suggest if there is any error. I am not from this field and trying TWARC by using tutorial. https://scholarslab.github.io/learn-twarc/07-utilities-mac

Thanks! That documentation definitely needs an update in that case. (For reference, this is the repo https://github.com/scholarslab/learn-twarc but i don't know who maintains it)

Also,

pip3 install networks

should be

pip3 install networkx

that would install the correct package, https://pypi.org/project/networkx/

[igorbrigadir]

Closing out some old issues: the solution for future is using plugins https://twarc-project.readthedocs.io/en/latest/plugins/ instead of the scripts. But to get older scripts in utils to run, the way to do it is to install the required packages manually. Unfortunately this is a bit of guesswork. Feel free to reopen / open a new issue if this comes up again.