Pystemon Problems

[Tango43]

Hi, i managed to install AIL-framework, but i am having problems figuring out how i activate the pystemon feeder to the AIL-framework, any help would be very helpful.
Thank you in advance

[Rafiot]

You have two possibilities:

• setting up pystemon
• Contact us and provide a static IP so we can give you access to our feed.

[Tango43]

Thank you for the answer.
I think using pystemon is the best option for my situation my question is how do I configure it and run it so that it feeds the information to AIL framework, because if I just start them at the same time no information seems to be reaching AIL-framework.

[Tango43]

I have now managed to store the pystemon files in AIL-framework/PASTES and to import them with import -d /home/"user"/AIL-framework/PASTES, but this forces me to import them every few minutes to analyse them is there a way to automate this process?

[Rafiot]

We have a feeder, @adulau willl give it to you :)

[Tango43]

I don't need it anyways I managed to get my feeder working :D
During the installation i got an error saying that jquery timers 1.0.0 path didn't exist anymore do you know how much this will impact the program? I can't find anywhere else i can download it from.

[Tango43]

--2016-06-01 12:25:52-- https://collabdev.googlecode.com/svn-history/r5/trunk/static/js/jquery.timers-1.0.0.js
Resolving collabdev.googlecode.com (collabdev.googlecode.com)... 64.233.166.82, 2a00:1450:400c:c09::52
Connecting to collabdev.googlecode.com (collabdev.googlecode.com)|64.233.166.82|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2016-06-01 12:25:52 ERROR 404: Not Found.

[Tango43]

[Rafiot]

yay, seems to work!

[Tango43]

My graph don't go down and look different from the ones on your pictures, could this be the jquery.timers ?

[Rafiot]

hmm, maybe. What happen if you reload the page?

[Tango43]

Nothing changes :/, also word trending doesn't work :P

[Rafiot]

I wonder where the jquery error comes from.

[Tango43]

Well to me it looks like the link is an old link and isn't valid anymore?

[Rafiot]

yeah, I didn't know we load external scripts...

Oh, ok, it is when you runthe update 3dr party script

[Tango43]

Yeah exactly

[Rafiot]

Can you try to get this one instead: https://raw.githubusercontent.com/jbrooksuk/jQuery-Timer-Plugin/master/jquery.timer.js ?

[Tango43]

Changed the lines in the update 3dr party script with the one u wrote, and then ran it again, but it didn't change anything still the exact same

[Rafiot]

are the values on the left side changing (going down)? And the logs (down the page)?

[Tango43]

Yes everything apart from the graphs and the wordtrending seems to be working as intended

[Rafiot]

Hmmmm this is super weird, the graph comes from the values on the left. Can you open a debug console in your browser and and see if there are errors?

[Tango43]

[Tango43]

Looks like that, btw sry I am just a intern that has been told to try this stuff out, not rly very comptetent with this stuff :D

[Tango43]

And wordstrending looks like this

[Rafiot]

Well, you got the framework too work, that's good :)

I just looked more closely at the 1st graph, it is actually moving up a bit. Could it be that you receive more pastes you can actually process? The values on the left, are they only increasing?

[Tango43]

Yeah, well the graphs only go up but never down even when i turn off the pastes, and even if i limit pystemon heavily to only giving me very few pastes (via the search feature) the graphs will only go up and never down

[Tango43]

I will try and increase my processing power on my VMmachine now

[Tango43]

It seems like to many things to process might have been the problem, they queues never went down cause i keept slaming them with the feeder, ups. Well ty for the help anyways was very helpful, just wordtrending not working now, but that seems like its a http path error as the server gives me 404 errors when i click it in its logs. I also checked the path ail-framework/var/www/static/csv/wordtrendingdata.csv and it is completely empty

[Rafiot]

ail-framework/var/www/static/csv/wordtrendingdata.csv <= could it be permission issue?

[Tango43]

Yeah it is the issue for some reason that .csv file is just not there?

[Rafiot]

it is created on the fly, by the web interface, I think.

[Tango43]

I should be launching Pystemon-feeder and Flask_server.py from within the virtual environment? the on that says
(AILENV)user@AILserver:

[Tango43]

This is what it looks like now some of queues seem to not get processed at all

[Rafiot]

To look at the scripts, and see if they are working, connect to the screen (screen -r)and switch between the windows to see what is happening (CTRL-a n)

[Tango43]

Yeah every one of them are having seperate errors. Will try and fix and then get back to you

[Rafiot]

if you fix the code, please do a Pull Request :)

[Tango43]

if not exists_in(indexpath):

File "/home/olivier/AIL-framework/AILENV/local/lib/python2.7/site-packages/whoosh/index.py", line 136, in exists_in
ix = open_dir(dirname, indexname=indexname)
File "/home/olivier/AIL-framework/AILENV/local/lib/python2.7/site-packages/whoosh/index.py", line 123, in open_dir
return FileIndex(storage, schema=schema, indexname=indexname)
File "/home/olivier/AIL-framework/AILENV/local/lib/python2.7/site-packages/whoosh/index.py", line 421, in init
TOC.read(self.storage, self.indexname, schema=self._schema)
File "/home/olivier/AIL-framework/AILENV/local/lib/python2.7/site-packages/whoosh/index.py", line 632, in read
check_size("int", _INT_SIZE)
File "/home/olivier/AIL-framework/AILENV/local/lib/python2.7/site-packages/whoosh/index.py", line 626, in check_size
sz = stream.read_varint()
File "/home/olivier/AIL-framework/AILENV/local/lib/python2.7/site-packages/whoosh/filedb/structfile.py", line 191, in read_varint
return read_varint(self.read)
File "/home/olivier/AIL-framework/AILENV/local/lib/python2.7/site-packages/whoosh/util/varints.py", line 102, in read_varint
b = ord(readfn(1))
TypeError: ord() expected a character, but string of length 0 found

[Tango43]

I keep getting this error for the indexer, are my python packages just installed wrongly? and how should i install them properly?

[Tango43]

Traceback (most recent call last):
File "./Curve.py", line 68, in
prev_score = r_serv1.hget(low_word, date)
File "/home/olivier/AIL-framework/AILENV/local/lib/python2.7/site-packages/redis/client.py", line 1857, in hget
return self.execute_command('HGET', name, key)
File "/home/olivier/AIL-framework/AILENV/local/lib/python2.7/site-packages/redis/client.py", line 578, in execute_command
connection.send_command(_args)
File "/home/olivier/AIL-framework/AILENV/local/lib/python2.7/site-packages/redis/connection.py", line 563, in send_command
self.send_packed_command(self.pack_command(_args))
File "/home/olivier/AIL-framework/AILENV/local/lib/python2.7/site-packages/redis/connection.py", line 538, in send_packed_command
self.connect()
File "/home/olivier/AIL-framework/AILENV/local/lib/python2.7/site-packages/redis/connection.py", line 442, in connect
raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error 111 connecting to localhost:2013. Connection refused.

[Rafiot]

Glad you solved the problem, and sorry I didn't follow up on this...