Problem getting data into skyline

[detzom]

Hi everyone,

I made and update on my skyline, and now data are no more integrated on redis.

I used a version of skyline downloaded by zip, dont know which revision exactly, but certainly the state of 20/02/2014 skyline-master.

when I start OLD horizon.d I have the following in log :

started with pid 2644
2014-06-16 11:45:10 :: 2644 :: starting horizon agent
2014-06-16 11:45:10 :: 2645 :: started worker
2014-06-16 11:45:10 :: 2644 :: started roomba
2014-06-16 11:45:10 :: 2648 :: started listener
2014-06-16 11:45:10 :: 2648 :: listening over udp for messagepack on 2025
2014-06-16 11:45:10 :: 2647 :: started listener
2014-06-16 11:45:10 :: 2647 :: listening over tcp for pickles on 2024
2014-06-16 11:45:10 :: 2646 :: started worker
2014-06-16 11:45:10 :: 2650 :: operated on metrics. in 0.235132 seconds
2014-06-16 11:45:10 :: 2650 :: metrics. keyspace is 1003
2014-06-16 11:45:10 :: 2650 :: blocked 0 times
2014-06-16 11:45:10 :: 2650 :: euthanized 0 geriatric keys
2014-06-16 11:45:10 :: 2650 :: sleeping due to low run time...
2014-06-16 11:45:11 :: 2647 :: connection from xxxxxxxxxxx:2024
2014-06-16 11:45:11 :: 2645 :: queue size at 134
2014-06-16 11:45:11 :: 2645 :: queue size at 326
2014-06-16 11:45:11 :: 2645 :: queue size at 488
...

when I start CURRENT horizon.d (with exactly same settings.py) I have the following in log :

started with pid 2702
2014-06-16 11:46:56 :: 2702 :: starting horizon agent
2014-06-16 11:46:56 :: 2703 :: started worker
2014-06-16 11:46:56 :: 2705 :: started listener
2014-06-16 11:46:56 :: 2702 :: started roomba
2014-06-16 11:46:56 :: 2705 :: listening over tcp for pickles on 2024
2014-06-16 11:46:56 :: 2706 :: started listener
2014-06-16 11:46:56 :: 2706 :: listening over udp for messagepack on 2025
2014-06-16 11:46:56 :: 2704 :: started worker
2014-06-16 11:46:57 :: 2708 :: operated on metrics. in 0.263879 seconds
2014-06-16 11:46:57 :: 2708 :: metrics. keyspace is 1007
2014-06-16 11:46:57 :: 2708 :: blocked 0 times
2014-06-16 11:46:57 :: 2708 :: euthanized 0 geriatric keys
2014-06-16 11:46:57 :: 2708 :: sleeping due to low run time...
2014-06-16 11:46:58 :: 2705 :: connection from xxxxxxxxxx:2024
2014-06-16 11:46:58 :: 2705 :: global name 'StringIO' is not defined
2014-06-16 11:46:58 :: 2705 :: incoming connection dropped, attempting to reconnect
2014-06-16 11:46:58 :: 2705 :: listening over tcp for pickles on 2024
2014-06-16 11:47:07 :: 2713 :: operated on metrics. in 0.252194 seconds
2014-06-16 11:47:07 :: 2713 :: metrics. keyspace is 1007
2014-06-16 11:47:07 :: 2713 :: blocked 0 times
2014-06-16 11:47:07 :: 2713 :: euthanized 0 geriatric keys
2014-06-16 11:47:07 :: 2713 :: sleeping due to low run time...
2014-06-16 11:47:11 :: 2704 :: worker queue is empty and timed out
2014-06-16 11:47:11 :: 2703 :: worker queue is empty and timed out
...

And no data is sent to redis.

Should I come back to my old working skyline or there is some changes on horizon.d that explain the change of behaviour concerning TCP PICKLES data transfers ?

[astanway]

On Mon, Jun 16, 2014 at 5:48 AM, detzom notifications@github.com wrote:

metrics. keyspace is 1007

You're sure no data is sent to Redis? Have you done a keys *? "metrics.
keyspace" is a direct measurement of the keys in Redis that were placed
there by the process.

I'm not sure what the StringIO is doing, it looks like it may be a bug with
the most recent commit (
1311d5b
)

Abe Stanway
abe.is

[detzom]

I checked my redis to ensure that no data is present :

started with pid 22763
2014-06-17 10:14:20 :: 22763 :: starting horizon agent
2014-06-17 10:14:20 :: 22764 :: started worker
2014-06-17 10:14:20 :: 22763 :: started roomba
2014-06-17 10:14:20 :: 22765 :: started worker
2014-06-17 10:14:20 :: 22766 :: started listener
2014-06-17 10:14:20 :: 22766 :: listening over tcp for pickles on 2024
2014-06-17 10:14:20 :: 22767 :: started listener
2014-06-17 10:14:20 :: 22767 :: listening over udp for messagepack on 2025
2014-06-17 10:14:20 :: 22769 :: operated on metrics. in 0.000259 seconds
2014-06-17 10:14:20 :: 22769 :: metrics. keyspace is 0
2014-06-17 10:14:20 :: 22769 :: blocked 0 times
2014-06-17 10:14:20 :: 22769 :: euthanized 0 geriatric keys
2014-06-17 10:14:20 :: 22769 :: sleeping due to low run time...
2014-06-17 10:14:22 :: 22766 :: connection from xxxxxxxxxxx:2024
2014-06-17 10:14:22 :: 22766 :: global name 'StringIO' is not defined
2014-06-17 10:14:22 :: 22766 :: incoming connection dropped, attempting to reconnect
2014-06-17 10:14:22 :: 22766 :: listening over tcp for pickles on 2024
2014-06-17 10:14:30 :: 22775 :: operated on metrics. in 0.000240 seconds
2014-06-17 10:14:30 :: 22775 :: metrics. keyspace is 0
2014-06-17 10:14:30 :: 22775 :: blocked 0 times
2014-06-17 10:14:30 :: 22775 :: euthanized 0 geriatric keys
2014-06-17 10:14:30 :: 22775 :: sleeping due to low run time...
2014-06-17 10:14:35 :: 22765 :: worker queue is empty and timed out
2014-06-17 10:14:35 :: 22764 :: worker queue is empty and timed out
2014-06-17 10:14:40 :: 22778 :: operated on metrics. in 0.001125 seconds
2014-06-17 10:14:40 :: 22778 :: metrics. keyspace is 0
2014-06-17 10:14:40 :: 22778 :: blocked 0 times
2014-06-17 10:14:40 :: 22778 :: euthanized 0 geriatric keys
2014-06-17 10:14:40 :: 22778 :: sleeping due to low run time...
2014-06-17 10:14:50 :: 22764 :: worker queue is empty and timed out
2014-06-17 10:14:50 :: 22765 :: worker queue is empty and timed out
2014-06-17 10:14:50 :: 22781 :: operated on metrics. in 0.000248 seconds
2014-06-17 10:14:50 :: 22781 :: metrics. keyspace is 0
2014-06-17 10:14:50 :: 22781 :: blocked 0 times
2014-06-17 10:14:50 :: 22781 :: euthanized 0 geriatric keys
2014-06-17 10:14:50 :: 22781 :: sleeping due to low run time...
2014-06-17 10:15:00 :: 22784 :: operated on metrics. in 0.000259 seconds
2014-06-17 10:15:00 :: 22784 :: metrics. keyspace is 0
2014-06-17 10:15:00 :: 22784 :: blocked 0 times
2014-06-17 10:15:00 :: 22784 :: euthanized 0 geriatric keys
2014-06-17 10:15:00 :: 22784 :: sleeping due to low run time...
2014-06-17 10:15:05 :: 22765 :: worker queue is empty and timed out
2014-06-17 10:15:05 :: 22764 :: worker queue is empty and timed out
2014-06-17 10:15:10 :: 22787 :: operated on metrics. in 0.000297 seconds
2014-06-17 10:15:10 :: 22787 :: metrics. keyspace is 0
2014-06-17 10:15:10 :: 22787 :: blocked 0 times
2014-06-17 10:15:10 :: 22787 :: euthanized 0 geriatric keys
2014-06-17 10:15:10 :: 22787 :: sleeping due to low run time...
[root@skyline bin]# redis-cli
127.0.0.1:6379> keys ?
(empty list or set)

And it seems not :)
StringIO is the only message between connection and connection drop of my carbon-relay server...

[astanway]

Are you sure you're sending data? Have you tried running seed_data.py?

On Jun 17, 2014, at 4:17 AM, detzom notifications@github.com wrote:

I checked my redis to ensure that no data is present :

started with pid 22763
2014-06-17 10:14:20 :: 22763 :: starting horizon agent
2014-06-17 10:14:20 :: 22764 :: started worker
2014-06-17 10:14:20 :: 22763 :: started roomba
2014-06-17 10:14:20 :: 22765 :: started worker
2014-06-17 10:14:20 :: 22766 :: started listener
2014-06-17 10:14:20 :: 22766 :: listening over tcp for pickles on 2024
2014-06-17 10:14:20 :: 22767 :: started listener
2014-06-17 10:14:20 :: 22767 :: listening over udp for messagepack on 2025
2014-06-17 10:14:20 :: 22769 :: operated on metrics. in 0.000259 seconds
2014-06-17 10:14:20 :: 22769 :: metrics. keyspace is 0
2014-06-17 10:14:20 :: 22769 :: blocked 0 times
2014-06-17 10:14:20 :: 22769 :: euthanized 0 geriatric keys
2014-06-17 10:14:20 :: 22769 :: sleeping due to low run time...
2014-06-17 10:14:22 :: 22766 :: connection from xxxxxxxxxxx:2024
2014-06-17 10:14:22 :: 22766 :: global name 'StringIO' is not defined
2014-06-17 10:14:22 :: 22766 :: incoming connection dropped, attempting to reconnect
2014-06-17 10:14:22 :: 22766 :: listening over tcp for pickles on 2024
2014-06-17 10:14:30 :: 22775 :: operated on metrics. in 0.000240 seconds
2014-06-17 10:14:30 :: 22775 :: metrics. keyspace is 0
2014-06-17 10:14:30 :: 22775 :: blocked 0 times
2014-06-17 10:14:30 :: 22775 :: euthanized 0 geriatric keys
2014-06-17 10:14:30 :: 22775 :: sleeping due to low run time...
2014-06-17 10:14:35 :: 22765 :: worker queue is empty and timed out
2014-06-17 10:14:35 :: 22764 :: worker queue is empty and timed out
2014-06-17 10:14:40 :: 22778 :: operated on metrics. in 0.001125 seconds
2014-06-17 10:14:40 :: 22778 :: metrics. keyspace is 0
2014-06-17 10:14:40 :: 22778 :: blocked 0 times
2014-06-17 10:14:40 :: 22778 :: euthanized 0 geriatric keys
2014-06-17 10:14:40 :: 22778 :: sleeping due to low run time...
2014-06-17 10:14:50 :: 22764 :: worker queue is empty and timed out
2014-06-17 10:14:50 :: 22765 :: worker queue is empty and timed out
2014-06-17 10:14:50 :: 22781 :: operated on metrics. in 0.000248 seconds
2014-06-17 10:14:50 :: 22781 :: metrics. keyspace is 0
2014-06-17 10:14:50 :: 22781 :: blocked 0 times
2014-06-17 10:14:50 :: 22781 :: euthanized 0 geriatric keys
2014-06-17 10:14:50 :: 22781 :: sleeping due to low run time...
2014-06-17 10:15:00 :: 22784 :: operated on metrics. in 0.000259 seconds
2014-06-17 10:15:00 :: 22784 :: metrics. keyspace is 0
2014-06-17 10:15:00 :: 22784 :: blocked 0 times
2014-06-17 10:15:00 :: 22784 :: euthanized 0 geriatric keys
2014-06-17 10:15:00 :: 22784 :: sleeping due to low run time...
2014-06-17 10:15:05 :: 22765 :: worker queue is empty and timed out
2014-06-17 10:15:05 :: 22764 :: worker queue is empty and timed out
2014-06-17 10:15:10 :: 22787 :: operated on metrics. in 0.000297 seconds
2014-06-17 10:15:10 :: 22787 :: metrics. keyspace is 0
2014-06-17 10:15:10 :: 22787 :: blocked 0 times
2014-06-17 10:15:10 :: 22787 :: euthanized 0 geriatric keys
2014-06-17 10:15:10 :: 22787 :: sleeping due to low run time...
[root@skyline bin]# redis-cli
127.0.0.1:6379> keys ?
(empty list or set)

And it seems not :)
StringIO is the only message between connection and connection drop of my carbon-relay server...

—
Reply to this email directly or view it on GitHub.

[detzom]

lol...

Yes im totally sure :) and my server is plugged on ;)

The last commit you pointed on seems to be the responsible, I reverted to previous one

[root@skyline bin]# git status
On branch master
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
nothing to commit (working directory clean)

And now :

started with pid 26713
2014-06-17 15:07:36 :: 26713 :: starting horizon agent
2014-06-17 15:07:36 :: 26714 :: started worker
2014-06-17 15:07:36 :: 26715 :: started worker
2014-06-17 15:07:36 :: 26716 :: started listener
2014-06-17 15:07:36 :: 26716 :: listening over tcp for pickles on 2024
2014-06-17 15:07:36 :: 26713 :: started roomba
2014-06-17 15:07:36 :: 26717 :: started listener
2014-06-17 15:07:36 :: 26717 :: listening over udp for messagepack on 2025
2014-06-17 15:07:36 :: 26719 :: operated on metrics. in 0.220659 seconds
2014-06-17 15:07:36 :: 26719 :: metrics. keyspace is 930
2014-06-17 15:07:36 :: 26719 :: blocked 0 times
2014-06-17 15:07:36 :: 26719 :: euthanized 0 geriatric keys
2014-06-17 15:07:36 :: 26719 :: sleeping due to low run time...
2014-06-17 15:07:38 :: 26716 :: connection from xxxxxxxxx:2024
2014-06-17 15:07:38 :: 26714 :: queue size at 17
2014-06-17 15:07:38 :: 26714 :: queue size at 415
2014-06-17 15:07:38 :: 26714 :: queue size at 472
2014-06-17 15:07:38 :: 26714 :: queue size at 466
2014-06-17 15:07:38 :: 26714 :: queue size at 451
2014-06-17 15:07:38 :: 26714 :: queue size at 465
2014-06-17 15:07:38 :: 26716 :: queue is full, dropping datapoints

127.0.0.1:6379> keys ?

1. "metrics.staging.app.banner.selector.isvalid"
2. "metrics.calcul.log1.stats.calcul-day.affichage.all-sum"
3. "metrics.staging.app.hit.conversion.valid.cookie"
4. "metrics.calcul.cl136.app.state.mysql"
5. "metrics.staging.app.log.file.today.bak"
6. "metrics.calcul.xxxxx.stats.calcul-memory.clic.all-count"
7. "metrics.calcul.xxxxx.app.log.file.today.coreg"
8. "metrics.calcul.log1.app.cron.diffusion.bid.running"
9. "metrics.calcul.cl136.app.madgicsocket.ms30_par_hit"
10. "metrics.calcul.xxxxxx.stats.export-newIndex-site_dayBilling.affichage.all-sum-affiliate"
11. "metrics.staging.app.debug.ifr.call_par_sec"
12. "metrics.calcul.xxxxxx.app.log.file.today.clicDefaut"
13. "metrics.staging.app.Redis.delete_par_sec"
14. "metrics.staging.app.log.file.today.error"
    ...

So im really sending data, but last commit has broken TCP PICKLE reception from carbon-relay.

[MyNameIsMeerkat]

This is my fault, missing dependencies. Fixing and committing now.

[MyNameIsMeerkat]

detzom, if you could check the fix is working for you that would be most appreciated and I will close

[detzom]

Fix is working great ;)

No more problem with last commit, thank you !

[MyNameIsMeerkat]

Awesomesauce