-
Notifications
You must be signed in to change notification settings - Fork 542
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Not thread safe? #42
Comments
Thread safety
Merge max length Solutions Alternatively there are secondary indexes supported in Cassandra (really hard to setup and maintain) Maybe you could store your own index in Redis so you only need two quick lookups. (though the 2 sequential lookups might be slower than just looping through 1000 rows, depends a bit on the size of the row and latency to redis) @tbarbugli more ideas? |
When i was testing it i was looking into the memory continuously. I have 14GiB of Memory and the operation never exceeded that. According to my knowledge feedly is not created with the thought that there should never be duplicate aggregated activities group, otherwise |
@Thierry where is the locking done at aggregated feed? I cant find it in @intellisense how do you regenerate feeds? The only problem I can think about the redis lock (as we use in the 2014-02-24 18:34 GMT+01:00 intellisense notifications@github.com:
|
Re: How do you regenerate feeds? P.S I am not spawning any extra celery tasks by myself. Celery tasks are those which feedly issues for |
I'm not sure we're talking about the same problem. Problem scenario A: Problem scenario B (Search depth) PS
|
Folks, you have the wrong 'Thierry' among the recipients. Could you please remove my name from this thread? Thanks in advance
|
Hi, sorry about that, they meant to contact me. |
I think only you can unsubscribe (there's a button) on the top right |
@tschellenbach I think its |
Any chance of a deadlock in using the above locking scheme? Because I can see 20%-30% celery fanout tasks are stuck forever while some succeeded. Using the locking technique is similar to celery Synchronous Subtasks Problem. Any suggestions? |
Hi! The redis lock clears out after the timeout. So it would need to go wrong Since it locks per feed and it processes one feed at the time deadlocks It is possible that a lock is set but not cleared. If that happens a task What Celery broker are you using? We've had many issues running anything
|
I am using redis as a broker. And redis comes from AWS Elastic Cache. Lock will be cleared automatically when exit from python |
Thats partially correct; killing the process that holds the locks will Regarding the issue, since locks are per feed, if you have lots of updates On Monday, 21 July 2014, intellisense notifications@github.com wrote:
sent from iphone (sorry for the typos) |
Hi, As written here
So I guess there is no work arround to add multiple user activities together in user aggregated feed and then fanout to followers? |
Hi, What backend storage are you using? 2014-08-16 23:14 GMT+02:00 intellisense notifications@github.com:
|
Just for testing i have set feed
max_length=1000
andmerge_max_length=1000
, why? because duplicate aggregated activities is not an option for me in newsfeed (see on facebook newsfeed you will never see a duplicate activity). So i ran a program which reads existing user actions from database and generates feeds using celery (async tasks).I was expecting that
max_length=merge_max_length
would never create a duplicate aggregated activity, but the program issue so many celery tasks that in the end there were some duplicate aggregated activities. Why that happen?Why there is even
merge_max_length
variable? I thought we were storing activities in redis as key, value where key is a group? so why there is a need to traverse through all aggregated activities to find where the new aggregated activity belong? A new aggregated activity already have group defined so why not do something like this as we do in pythondict.get(group)
?The text was updated successfully, but these errors were encountered: