fakeredis can't work with resque scheduler #7

Closed
saberma opened this Issue Dec 1, 2011 · 9 comments

Comments

Projects
None yet
4 participants

saberma commented Dec 1, 2011

QUEUE=* bundle exec rake resque:scheduler

rake aborted!
undefined method `zrangebyscore' for #<Redis::Connection::Memory:0xb92ac38>
/home/saberma/.rvm/gems/ruby-1.9.3-p0/gems/fakeredis-0.2.2/lib/redis/connection/memory.rb:93:in `write'
/home/saberma/.rvm/gems/ruby-1.9.3-p0/gems/redis-2.2.2/lib/redis/client.rb:139:in `block (3 levels) in process'
/home/saberma/.rvm/gems/ruby-1.9.3-p0/gems/redis-2.2.2/lib/redis/client.rb:138:in `each'
/home/saberma/.rvm/gems/ruby-1.9.3-p0/gems/redis-2.2.2/lib/redis/client.rb:138:in `block (2 levels) in process'
/home/saberma/.rvm/gems/ruby-1.9.3-p0/gems/redis-2.2.2/lib/redis/client.rb:250:in `ensure_connected'
/home/saberma/.rvm/gems/ruby-1.9.3-p0/gems/redis-2.2.2/lib/redis/client.rb:137:in `block in process'
/home/saberma/.rvm/gems/ruby-1.9.3-p0/gems/redis-2.2.2/lib/redis/client.rb:206:in `logging'
/home/saberma/.rvm/gems/ruby-1.9.3-p0/gems/redis-2.2.2/lib/redis/client.rb:136:in `process'
/home/saberma/.rvm/gems/ruby-1.9.3-p0/gems/redis-2.2.2/lib/redis/client.rb:46:in `call'
/home/saberma/.rvm/gems/ruby-1.9.3-p0/gems/redis-2.2.2/lib/redis.rb:573:in `block in zrangebyscore'
/home/saberma/.rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
/home/saberma/.rvm/gems/ruby-1.9.3-p0/gems/redis-2.2.2/lib/redis.rb:572:in `zrangebyscore'
/home/saberma/.rvm/gems/ruby-1.9.3-p0/gems/redis-namespace-1.0.3/lib/redis/namespace.rb:213:in `method_missing'
/home/saberma/.rvm/gems/ruby-1.9.3-p0/bundler/gems/resque-scheduler-54075b0f5492/lib/resque_scheduler.rb:170:in `next_delayed_timestamp'
/home/saberma/.rvm/gems/ruby-1.9.3-p0/bundler/gems/resque-scheduler-54075b0f5492/lib/resque/scheduler.rb:147:in `handle_delayed_items'
/home/saberma/.rvm/gems/ruby-1.9.3-p0/bundler/gems/resque-scheduler-54075b0f5492/lib/resque/scheduler.rb:51:in `block in run'
/home/saberma/.rvm/gems/ruby-1.9.3-p0/bundler/gems/resque-scheduler-54075b0f5492/lib/resque/scheduler.rb:49:in `loop'
/home/saberma/.rvm/gems/ruby-1.9.3-p0/bundler/gems/resque-scheduler-54075b0f5492/lib/resque/scheduler.rb:49:in `run'
/home/saberma/.rvm/gems/ruby-1.9.3-p0/bundler/gems/resque-scheduler-54075b0f5492/lib/resque_scheduler/tasks.rb:23:in `block (2 levels) in <top (required)>'

Viximo/fakeredis@7f81b71

This guy implement the method, consider to merge it?

Owner

guilleiguaran commented Dec 1, 2011

I merged #8, can you test if resque scheduler is working with it?

saberma commented Dec 1, 2011

ok, I get the latest fakeredis from github directly, but it raise another error:

redis.zrangebyscore :delayed_queue_schedule, '-inf', Time.now.to_i, :limit => [0, 1]
ArgumentError: wrong number of arguments (6 for 4)
    from /home/saberma/.rvm/gems/ruby-1.9.3-p0/bundler/gems/fakeredis-fa2ee288152c/lib/redis/connection/memory.rb:762:in `zrangebyscore'
    from /home/saberma/.rvm/gems/ruby-1.9.3-p0/bundler/gems/fakeredis-fa2ee288152c/lib/redis/connection/memory.rb:97:in `write'
    from /home/saberma/.rvm/gems/ruby-1.9.3-p0/gems/redis-2.2.2/lib/redis/client.rb:139:in `block (3 levels) in process'
    from /home/saberma/.rvm/gems/ruby-1.9.3-p0/gems/redis-2.2.2/lib/redis/client.rb:138:in `each'
    from /home/saberma/.rvm/gems/ruby-1.9.3-p0/gems/redis-2.2.2/lib/redis/client.rb:138:in `block (2 levels) in process'
    from /home/saberma/.rvm/gems/ruby-1.9.3-p0/gems/redis-2.2.2/lib/redis/client.rb:250:in `ensure_connected'
    from /home/saberma/.rvm/gems/ruby-1.9.3-p0/gems/redis-2.2.2/lib/redis/client.rb:137:in `block in process'
    from /home/saberma/.rvm/gems/ruby-1.9.3-p0/gems/redis-2.2.2/lib/redis/client.rb:206:in `logging'
    from /home/saberma/.rvm/gems/ruby-1.9.3-p0/gems/redis-2.2.2/lib/redis/client.rb:136:in `process'
    from /home/saberma/.rvm/gems/ruby-1.9.3-p0/gems/redis-2.2.2/lib/redis/client.rb:46:in `call'
    from /home/saberma/.rvm/gems/ruby-1.9.3-p0/gems/redis-2.2.2/lib/redis.rb:573:in `block in zrangebyscore'
    from /home/saberma/.rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
    from /home/saberma/.rvm/gems/ruby-1.9.3-p0/gems/redis-2.2.2/lib/redis.rb:572:in `zrangebyscore'

It's strange that :limit => [0, 1] was recognize as three params?

Collaborator

caius commented Feb 21, 2013

Not sure if this is still an issue for you @saberma, but as a FYI, the work in master in the last few months may well have fixed this issue.

same issue here :(

How to reproduce the problem:

  • create a Rails project which add jobs to a resque queue, like this: http://railscasts.com/episodes/271-resque
  • open a shell terminal and start the server (rails s)
  • open another shell terminal and start a resque worker (rake resque:work QUEUE='*')
  • on the browser, create a new job

What is expected?

  • the jobs are processed

What occours?

  • nothing. No errors at terminals ("rails s" / "rake resque:work"). The status of the jobs at resque table aren't modified by the worker. It's like if the "rake resque:work" aren't running.

Why?
after a quick look at the source-code, I guess that this is because fakeredis are running at memory, so different process (rails server vs rake) cannot share the data. If this is true, many popular gems (ohm database, resque, sidekiq, etc.) will not work.

Possible simple solution
use "Rails.cache" instead of memory.

Collaborator

caius commented Oct 19, 2013

@loureirorg I'm not sure fakeredis will really solve the issue you have here. Sounds more like you need an actual redis server running to talk to, which will allow multiple processes to share data through it. Fakeredis is intended for mocking out a redis server in a test suite run for example, where there is only one process and speed is required rather than an actual redis server with all the features that brings.

good question @caius. I don't know very much about the project, in fact I discover it yesterday, but I was thinking that the fakeredis was the sqlite equivalent for the redis. And if is not, why don't be?

Look at the sqlite: who uses it? Why use it? It terrible slow, and don't manage well even a razonable quantity of data. But it still used, and is a valuable tool. Why? There are 2 main uses: a) you have a little ammount of data, which will not be accessed many times (so be slow it's not a problem), just by one machine, and you want to have the sql power to manage this data - in this case sqlite will be a good option even in production (firefox is an example). In this case a whole DBMS server, if used, would be underutilized. b) you are in development (eg. rails development) and:

  • you want to start your app without start any other servers - lazy developers are efficient developers ;)
  • you want to copy your entire project folder to a pendrive and use, with the same data, in other machine
  • you don't want to install and start a dbms server (it's ridiculously easy, but lazyness win again)
  • you will upload this test-project to a host server (heroku, getupcloud, amazon, digitalocean, etc), but you don't want to know how to set up a database at this specific host-server - and you even want to pay for this additional feature
  • you know that when you put this in production you will have to use a dbms server, but you don't want to change the api calls in the code - so you need to use the same api (sql language / methods to call) for the dbms who is in production and the local-emulated dbms who is in development
  • you don't want to mix data of many projects (probally they use the same names for tables/database)
  • and many others

There's a plenty of reasons to why use a local sql database, in fact they are so strong that the rails community includes the sqlite by default in any new projects (and I guess that the lazyness was the main reason, if not the only). But this is with sql, which is basic for virtually any web application, so what about the redis? Today key-hash-stores are almost onipresent like sql-stores. Ohm (active record using redis), resque, sidekiq, and many others. Just here in github are more than 1000 projects with "ruby redis" tag. You can even, and use its powerful api to manage data inside your internal app. And in this scenario redis is becoming the default interface - the sql equivalent of the dbms.

Maybe this aren't the main goal of fakeredis (to be an sqlite equivalent of the redis interface), but I believe that is a good way to go because it take the same users, which the same needs. Why you want to use fakeredis? Is not by the same questions that guide a sqlite choice? The very same above? And the actual stage of this project satisfies all those needs, but don't permit share the data across processes neither persists its data (the data is destroyed at end of the process because is on memory).

But, this is just my two cents to this question ;)

Collaborator

caius commented Oct 19, 2013

@loureirorg I think you might be misunderstanding the aims of this project slightly, and how it relates to redis' aims. Redis is already an in-memory key value datastore, and what I would consider to be akin to the SQLite of the no-sql database world already. The downside is when you build an application against it, you want to run your test suite against it, and there is some overhead of talking to an external system (to say nothing of requiring someone to have a local redis server running just to run your tests.) Fakeredis (as I understand it - happy to be corrected by @guilleiguaran if I'm wrong here) aims to solve that latter problem, by providing an in-process (and therefore in-memory) functional mock of redis, so the test suite doesn't have to talk to the external redis server.

Going down the route of reimplementing the datastore under Rails.cache is starting to veer too much towards reimplementing redis badly in ruby, and really wouldn't be feasible or performant compared to just using redis. Redis has had lots of clever optimisations applied, and has been designed from the start to be fairly simple in terms of protocol and what it does, but damn quick at what it does. Fakeredis just aims to be quicker within the same process, for usage in a test suite (for example).

I would strongly suggest you go take redis for a spin instead of fakeredis and see just how awesome it is. And then think of fakeredis if you write tests and want to remove the reliance on having your tests call out to an external service. (It should be as simple as installing redis-server on your machine, having the redis gem in your gemfile, and removing the fakeredis gem.) You should find all your resque workers can see the jobs your rails processes are placing into redis then. Enjoy!

thanks for the response. So, this issue will never marked as "resolved", cause is not a bug. I believe that you can mark as "wontfix". I made a fork called redis-file for those who have the same needs that me. If you can help to divulge for those who have the same mistake that me, I would appreciate.

Collaborator

caius commented May 20, 2014

I'm sorry fakeredis didn't quite fit your needs, but I'll keep redis-file in mind to recommend if anyone ever asks for the same sort of solution in the future. Good luck, and I hope you find the solution you need in redis-file! 😀

@caius caius closed this May 20, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment