Configuration & Setup & Flask-Server

[nbdy]

I might be just stupid..
Backstory:
I cloned the repository.
Installed all dependencies.

Path 1:
Ran the AIL-Framework.
With:
LevelDB
Redis
Logs
Queues
Scripts

Ran pystemon with my own configured yaml.
Ran the Flask-Server.
Nothing except the main interface. No data, no listing, nothing.

Path 2:
Set the path in the pystemon-feeder script to '/opt/pystemon/' (where pystemon is located)
Ran the feeder
Ran the AIL-Framework with the options in path 1
Ran the Flask-server.
Again: Nothing except the main UI.

Did I miss to configure something?
Also:
I can't kill the Flask-Server with CTRL+C, it just gets unresponsive.
And the AILENV variable needs to be reset after reboot.

Running on Ubuntu 14.04 LTS without xServer (plain cli rootserver)

[Beercow]

This threw me off for a while too. It is not well documented. The reason why everthing is blank is because you need to edit a cofig file. In the AIL-framework/bin/packages/ folder edit config.cfg and config.cfg.sample. Find the line adress = tcp://crf.circl.lu:5556 and change it to adress = tcp://127.0.0.1:5556. When pystemon find something, it should now show up in the web GUI.

[nbdy]

Thanks for the fast response.
I've just edited those 2 files.
Do I need to run the feeder seperatly?
Since there is still nothing coming in. (Might be because I'm on my phone, but there should be at least entrys in the table on the lower of the page.)

[Beercow]

The feeder script needs to be running and needs to be edited to point to your pystemon location. Launch the AIL-framework, start the feeder script, start pystemon, and launch the flask server. Check all screens and make sure there are no errors. For testing, I edited the pystemon.yaml file to look for any email addresses. You should start seeing logs and graph activity.

[Beercow]

Forgot to mention. In the pystemon.yaml, make sure redis is set to yes.

[nbdy]

Ok, I thought the feeder runs when I start the whole framework.
The Path is edited and redis is set to 'yes' in the yaml.
Upto now nothing happend.
Might need to restart it.
Is there a certain order I have to launch it in?
Btw. I'm using tmux instead of screen.
If you say it should definietly work now, I would just re-setup that whole thing and try again.

[nbdy]

Little Update:
Still does not show anything.
Current config: http://pastebin.com/kfpsi2q1
The redis is set to yes. In the pastebin link it's set to no, since I copied it from my issue from pystemon.
Also tried to change the Redis DB to 10 in packages/config.cfg
Anything else I did miss?
(Might want to set it up completely manually.

[Beercow]

I think the problem might be your pystemon.yaml file. The search fields for the regex should have the dash in front of them like the description does or it won't work. example:

• search: '[A-Z0-9_.+-]+@[A-Z0-9.-]+.[A-Z]{2,4}'

this would search for anything with a email address.

[Beercow]

Make sure to set the DB setting back to what it was in the config.cfg file.

[Beercow]

You can try my fork of pystemon, if you want. It scrapes 26 different sites. Just place your regex searches in the yaml file and change the the section under proxy: from use: yes to use: no.

https://github.com/Beercow/pystemon.git

[nbdy]

I'm going to re-setup my whole server, doing everything from scratch with your pystemon fork. (Seems like you are pretty good with regexes.)
But that will probably take until tomorrow.
Thanks for all the help though!
OT: For what are you using the local proxy? (What kinda advantage gives that you?) If I may ask.

[Beercow]

I did the automatic setup. Change the path in the feeder script and the IP in the cfg file and change the proxy setting to no in the yaml and use the regex search I posted earlier for testing (you should get alot of hits). As for a proxy, I am using TOR. The advantage to it is that your IP changes every 10 minutes. This helps to not get blocked by sites like pastebin. Here is a link to the article I used to set up TOR. https://mendo.pt/monitoring-pastebin/ . Get everything else working before taking the next step and trying to run it through a proxy. Let me know how it turns out or if you need any help. It took me a while to figure all this out on my own.

[nbdy]

Well. Little update:
Fresh installed Ubuntu 14.04 LTS
Did everything as described above.
Pystemon now shows hits.
Got the feeder running in a tmux session.
Got the flask server running in a tmux session.
Got pystemon running in a tmux session.
Still no output on the Website.
Don't really know anymore.
Slightly intetesting response in the flask server output:
/static//csv/wordstrendingdata.csv HTTP 1.1 404

[Beercow]

Permissions issue maybe? I also run pystemon from the AILENV. Try running everything as root and see what happens.

[Beercow]

Did you check the screen sessions for leveldb, redis, logs, scripts, and queues for any errors? Also netstat -plnt to make sure everything is listening on the ports.

[nbdy]

I'm nearly always running everything as root.(Not really secure, I know.)
Tried to launch pystemon from AILENV.
It asks for the BeautifulSoup3 Library.
Installed it via apt-get and pip. Still asks for it.
Got python listening on 5556 (probably the feeder)
Redis leveldb on 2013 and 2014
Redis Server on 6379 and 6380 and 6381
Python on 5004 and 5005?
Looks how it should to me.

No errors, is it normal that they have no output?

[nbdy]

I'm closing this for now.
Rebooted my server.
Killed all redis processes. I found with netstat -tulpn.
Started the everything from the LAUNCH.sh.
Started the feeder.
Started pystemon.
It somehow magically worked, and I get hits.
Don't really know what solved the problem.
Seems like it's the magic of rebooting.
Thanks alot for your help!