New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Weird recommendation behavior #91

Open
hoguej opened this Issue Apr 24, 2014 · 19 comments

Comments

Projects
None yet
2 participants
@hoguej

hoguej commented Apr 24, 2014

When I dislike a course, it recommends it to similar users.

class user < ActiveRecord::Base
  has_many :courses
  recommends :course
end

user1.like(Course.find(2622))
user1.like(Course.find(2431))
user1.like(Course.find(1466))
user1.dislike(Course.find(315))

user2.like(Course.find(2622))
user2.like(Course.find(2431)
user2.recommended_courses.map{ |c| c.id }
 => [315, 1466]

user2.liked_course_ids
=> [2431, 2622]

user1.liked_course_ids
 => [1466, 2431, 2622]

user1.disliked_course_ids
 => [315]
@davidcelis

This comment has been minimized.

Show comment
Hide comment
@davidcelis

davidcelis Apr 24, 2014

Owner

Ah, yeah. The reason for this is that all recommendation values (or the top N values if you've used config.nearest_neighbors) get stored in Redis. So if you've only been compared with a few other users, it's possible that your top recommendations would include a recommendation with a score of less than 0 (which happens when similar users dislike that item). I wonder if it would be best to exclude any item with a negative score from returned recommendations?

Owner

davidcelis commented Apr 24, 2014

Ah, yeah. The reason for this is that all recommendation values (or the top N values if you've used config.nearest_neighbors) get stored in Redis. So if you've only been compared with a few other users, it's possible that your top recommendations would include a recommendation with a score of less than 0 (which happens when similar users dislike that item). I wonder if it would be best to exclude any item with a negative score from returned recommendations?

@hoguej

This comment has been minimized.

Show comment
Hide comment
@hoguej

hoguej Apr 24, 2014

Should at least be sorted by highest score, right? 315 should have a negative score, and 1466 is positive. You'd expect 1466 to be first?

hoguej commented Apr 24, 2014

Should at least be sorted by highest score, right? 315 should have a negative score, and 1466 is positive. You'd expect 1466 to be first?

@hoguej

This comment has been minimized.

Show comment
Hide comment
@hoguej

hoguej Apr 24, 2014

But would also make sense to remove negative scores from recommended_(). maybe expose that through a new method not_recommended_ or something.

hoguej commented Apr 24, 2014

But would also make sense to remove negative scores from recommended_(). maybe expose that through a new method not_recommended_ or something.

@davidcelis

This comment has been minimized.

Show comment
Hide comment
@davidcelis

davidcelis Apr 24, 2014

Owner

It's definitely sorted by highest score. Are you sure that 315 has a negative score, or a lower score than 1466? I'd inspect the Redis ZSET directly. Are there other ratings at work here?

Owner

davidcelis commented Apr 24, 2014

It's definitely sorted by highest score. Are you sure that 315 has a negative score, or a lower score than 1466? I'd inspect the Redis ZSET directly. Are there other ratings at work here?

@hoguej

This comment has been minimized.

Show comment
Hide comment
@hoguej

hoguej Apr 25, 2014

recommendable:users:134255:recommended_courses

value score
315 -1
1466 1

hoguej commented Apr 25, 2014

recommendable:users:134255:recommended_courses

value score
315 -1
1466 1

@hoguej

This comment has been minimized.

Show comment
Hide comment
@hoguej

hoguej Apr 25, 2014

I'm not sure what you mean by ratings, exactly.

Checking redis, it seems like the data is consistent with what I'd expect from the steps I followed in the first post.

hoguej commented Apr 25, 2014

I'm not sure what you mean by ratings, exactly.

Checking redis, it seems like the data is consistent with what I'd expect from the steps I followed in the first post.

@hoguej

This comment has been minimized.

Show comment
Hide comment
@hoguej

hoguej Apr 25, 2014

So, my course class has a rated? method, unrelated to recommendable. Wonder if that's causing a problem.

hoguej commented Apr 25, 2014

So, my course class has a rated? method, unrelated to recommendable. Wonder if that's causing a problem.

@hoguej

This comment has been minimized.

Show comment
Hide comment
@hoguej

hoguej Apr 25, 2014

Rename that method and reran everything. Same result. I guess next step is to create a minimal example app and try to reproduce the problem there.

hoguej commented Apr 25, 2014

Rename that method and reran everything. Same result. I guess next step is to create a minimal example app and try to reproduce the problem there.

@davidcelis

This comment has been minimized.

Show comment
Hide comment
@davidcelis

davidcelis Apr 25, 2014

Owner

Sorry, typoed that. I meant to ask if there are any other ratings (likes/dislikes) at play here.

Owner

davidcelis commented Apr 25, 2014

Sorry, typoed that. I meant to ask if there are any other ratings (likes/dislikes) at play here.

@davidcelis

This comment has been minimized.

Show comment
Hide comment
@davidcelis

davidcelis Apr 25, 2014

Owner

Anyway, please do try to reproduce it in a codebase I can clone down and take a look at. Thanks!

Owner

davidcelis commented Apr 25, 2014

Anyway, please do try to reproduce it in a codebase I can clone down and take a look at. Thanks!

@hoguej

This comment has been minimized.

Show comment
Hide comment
@hoguej

hoguej Apr 25, 2014

I've created a simple sample app, but noticed something that is probably the culprit.

Notice, that we look for (5,3,4) but we get back [3,4,5]. This is rails 3. Not sure if rails 4 handles this more appropriately.

1.9.3-p484 :001 > user2 = User.find(2); user2.recommended_courses.map{ |c| c.id }
User Load (2.9ms) SELECT "users".* FROM "users" WHERE "users"."id" = ? LIMIT 1 [["id", 2]]
Course Load (0.1ms) SELECT "courses".* FROM "courses" WHERE "courses"."id" IN (5, 3, 4)
=> [3, 4, 5]
1.9.3-p484 :002 >

hoguej commented Apr 25, 2014

I've created a simple sample app, but noticed something that is probably the culprit.

Notice, that we look for (5,3,4) but we get back [3,4,5]. This is rails 3. Not sure if rails 4 handles this more appropriately.

1.9.3-p484 :001 > user2 = User.find(2); user2.recommended_courses.map{ |c| c.id }
User Load (2.9ms) SELECT "users".* FROM "users" WHERE "users"."id" = ? LIMIT 1 [["id", 2]]
Course Load (0.1ms) SELECT "courses".* FROM "courses" WHERE "courses"."id" IN (5, 3, 4)
=> [3, 4, 5]
1.9.3-p484 :002 >

@hoguej

This comment has been minimized.

Show comment
Hide comment
@hoguej

hoguej Apr 25, 2014

I do something like this to deal with this same situation from solr. Not the most efficient, but database agnostic. If I could access the raw scores, I could do something like that myself.

courses.sort!{ |x,y| course_ids.index(x.id.to_s) <=> course_ids.index(y.id.to_s) }

hoguej commented Apr 25, 2014

I do something like this to deal with this same situation from solr. Not the most efficient, but database agnostic. If I could access the raw scores, I could do something like that myself.

courses.sort!{ |x,y| course_ids.index(x.id.to_s) <=> course_ids.index(y.id.to_s) }

@davidcelis

This comment has been minimized.

Show comment
Hide comment
@davidcelis

davidcelis Apr 25, 2014

Owner

Shoot, you're right: https://github.com/davidcelis/recommendable/blob/master/lib/recommendable/rater/recommender.rb#L21-L27

I thought that I was correctly sorting the records before returning them to the user, but that's not the case. So while that list does contain the best recommendations, they're not sorted within that list. I probably did this originally to keep that query chainable, but I'm willing to wager that it's more important for that list to be in the right order.

Owner

davidcelis commented Apr 25, 2014

Shoot, you're right: https://github.com/davidcelis/recommendable/blob/master/lib/recommendable/rater/recommender.rb#L21-L27

I thought that I was correctly sorting the records before returning them to the user, but that's not the case. So while that list does contain the best recommendations, they're not sorted within that list. I probably did this originally to keep that query chainable, but I'm willing to wager that it's more important for that list to be in the right order.

@hoguej

This comment has been minimized.

Show comment
Hide comment
@hoguej

hoguej Apr 25, 2014

So while that list does contain the best recommendations
Except that 4 should not show up at all since it was disliked. At most, it should show up last. Because the negative numbers are in there, and it's out of order, it's giving the wrong answer.

Also, seems weird it would even be in the return result, since there's no way to know where the positive stops and the negative starts. You could argue, with a fully populated recommendation engine, it would probably give the correct result. Might be true where you are getting 10 results back, but false if getting back 100's of recommendations.

I can take a crack at a patch, but I'm new to this lib and redis.

hoguej commented Apr 25, 2014

So while that list does contain the best recommendations
Except that 4 should not show up at all since it was disliked. At most, it should show up last. Because the negative numbers are in there, and it's out of order, it's giving the wrong answer.

Also, seems weird it would even be in the return result, since there's no way to know where the positive stops and the negative starts. You could argue, with a fully populated recommendation engine, it would probably give the correct result. Might be true where you are getting 10 results back, but false if getting back 100's of recommendations.

I can take a crack at a patch, but I'm new to this lib and redis.

@davidcelis

This comment has been minimized.

Show comment
Hide comment
@davidcelis

davidcelis Apr 25, 2014

Owner

Good point. I think that at the very least, items with negative scores should never be recommended. I can definitely fix that much, but I'll need to think a bit more on sacrificing the chainability of querying against those recommendations for having them ordered correctly. There is a tradeoff there, but I'm leaning towards ordering them correctly.

Owner

davidcelis commented Apr 25, 2014

Good point. I think that at the very least, items with negative scores should never be recommended. I can definitely fix that much, but I'll need to think a bit more on sacrificing the chainability of querying against those recommendations for having them ordered correctly. There is a tradeoff there, but I'm leaning towards ordering them correctly.

@hoguej

This comment has been minimized.

Show comment
Hide comment
@hoguej

hoguej Apr 25, 2014

Wonder if we can dip into the ActiveRecord class to lazily store the order.

On Fri, Apr 25, 2014 at 3:26 PM, David Celis notifications@github.comwrote:

Good point. I think that at the very least, items with negative scores
should never be recommended. I can definitely fix that much, but I'll need
to think a bit more on sacrificing the chainability of querying against
those recommendations for having them ordered correctly. There is a
tradeoff there, but I'm leaning towards ordering them correctly.

Reply to this email directly or view it on GitHubhttps://github.com//issues/91#issuecomment-41430409
.

hoguej commented Apr 25, 2014

Wonder if we can dip into the ActiveRecord class to lazily store the order.

On Fri, Apr 25, 2014 at 3:26 PM, David Celis notifications@github.comwrote:

Good point. I think that at the very least, items with negative scores
should never be recommended. I can definitely fix that much, but I'll need
to think a bit more on sacrificing the chainability of querying against
those recommendations for having them ordered correctly. There is a
tradeoff there, but I'm leaning towards ordering them correctly.

Reply to this email directly or view it on GitHubhttps://github.com//issues/91#issuecomment-41430409
.

@davidcelis

This comment has been minimized.

Show comment
Hide comment
@davidcelis

davidcelis May 18, 2014

Owner

@hoguej I decided to abandon leaving the recommended_things methods chainable so that recommendations are sorted correctly. This has landed as of a0e77bd. I'll leave this open if you want to confirm that it works.

Owner

davidcelis commented May 18, 2014

@hoguej I decided to abandon leaving the recommended_things methods chainable so that recommendations are sorted correctly. This has landed as of a0e77bd. I'll leave this open if you want to confirm that it works.

@hoguej

This comment has been minimized.

Show comment
Hide comment
@hoguej

hoguej May 19, 2014

I'll take a look this week. Thanks for the update. When I get to it, I'll expose my test app.

hoguej commented May 19, 2014

I'll take a look this week. Thanks for the update. When I get to it, I'll expose my test app.

@hoguej

This comment has been minimized.

Show comment
Hide comment
@hoguej

hoguej May 27, 2014

Sorry, got ill last week. Still on my agenda.

hoguej commented May 27, 2014

Sorry, got ill last week. Still on my agenda.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment