New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize `CoursesController#index` #890

Merged
merged 1 commit into from Apr 14, 2018

Conversation

2 participants
@chancancode
Contributor

chancancode commented Apr 14, 2018

This is a follow up to #888 (I also work at Tilde/Skylight).

Currently, a significant amount of time (up to 40%!) is spent in loading and initialize the lesson objects:

skylight

We originally suspected a missing index, but as @gitKrystan and @zvkemp pointed out, this wasn't the case. Turns out, the reason it is slow is because the lessons have a big content field, making them very expensive to load:

  1. The database has to read a lot into memory
  2. The database has to send back a big result
  3. The Ruby driver has to parse/process the big payload
  4. The content field need to be loaded/copied into big Ruby strings
  5. Having these big strings in memory increases memory usage/GC pressure

Therefore, it is a good idea to avoid loading the lesson objects unless they are absolutely needed to display the content, which is probably only needed in LessonsController#show.

In the case of CoursesController#index, the lesson objects are only loaded to render the badges. This PR demonstrates a way to refactor the relevant code to avoid
needing to load the lessons object.

The focus here is CoursesController#index, but a lot of other endpoints can benefit from this optimization (you can find them on Skylight). I'll leave that as an exercise to interested readers 馃槈

I ran a synthetic benchmark locally:

  • I ran rails server -e productionlocally, while sending data to a fresh Skylight app
  • I used ab to send 1000 requests to /courses, before and after the patch, with and without logging in

The results look quite promising, ranging from 11-40% improvement:

skylight

I would expect to see a bigger improvement on the actual production environment:

  1. The synthetic benchmark access the same data/endpoint repeatedly in a short amount of time, making it a best case scenario for the database (things are already in working memory, etc)
  2. The database and web server is on the same computer, eliminating time to send those big blobs across the network
  3. My database only has the seed data (and a few rows of lesson_completions for a single user)
  4. For the same reason as (1), the Rails app probably has much lower GC pressure than production

If this patch is accepted, I have some follow up suggestions for others who are interested in doing performance work:

  1. Do the same work to avoid the need to load the lessons object for other endpoints (CoursesController#show, etc).
  2. Remove @user and consistently use current_user everywhere. As far as I can tell, this was only added to force an eager load on lesson_completions. I believe this is no longer necessary (and in fact it is now doing an extra/redundant query).
  3. Refactor: merge NextLesson into CourseProgress#next_lesson to remove some duplication and take advantage of better caching.
  4. Consider denormalizing lessons, i.e. add a course_id column, an index on the column, and belongs_to :course to the lessons table/model (and possibly lesson_completions as well). It is not rare that we want to get a list of lessons associated with a course (which I had to do a few times here), but since the only way to do that is to go through sections today, fetching the lessons for a course requires a somewhat expensive JOIN. Having the course_id on the lessons object would help eliminate that cost.

Finally, it is also worth noting: the "logged in" scenario is not as optimized as it could be. If you look at Skylight or the logs, you will see something like this:

[9ba5c805-806c-4ff5-9d5e-48fa283a2be8] Started GET "/courses" for 127.0.0.1 at 2018-04-13 21:30:07 -0700
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8] Processing by CoursesController#index as HTML
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]   Parameters: {}
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]   User Load (1.6ms)  SELECT  "users".* FROM "users" ORDER BY "users"."id" ASC LIMIT $1  [["LIMIT", 1]]
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]   Course Load (0.8ms)  SELECT "courses".* FROM "courses" ORDER BY "courses"."position" ASC
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]   Rendering layouts/application.html.erb
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]   Rendering courses/index.html.erb within layouts/application
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]    (1.1ms)  SELECT "lessons".id FROM "lessons" INNER JOIN "sections" ON "lessons"."section_id" = "sections"."id" WHERE "sections"."course_id" = $1 ORDER BY "lessons"."position" ASC, "sections"."position" ASC  [["course_id", 1]]
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]    (0.7ms)  SELECT COUNT(*) FROM "lesson_completions" WHERE "lesson_completions"."student_id" = $1 AND "lesson_completions"."lesson_id" IN (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35)  [["student_id", 1]]
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]   Rendered shared/_course_badge.html.erb (0.7ms)
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]    (0.9ms)  SELECT COUNT(*) FROM "lessons" INNER JOIN "sections" ON "lessons"."section_id" = "sections"."id" WHERE "sections"."course_id" = $1  [["course_id", 1]]
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]   Rendered courses/_course_card.html.erb (99.0ms)
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]    (0.6ms)  SELECT "lessons".id FROM "lessons" INNER JOIN "sections" ON "lessons"."section_id" = "sections"."id" WHERE "sections"."course_id" = $1 ORDER BY "lessons"."position" ASC, "sections"."position" ASC  [["course_id", 2]]
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]    (0.4ms)  SELECT COUNT(*) FROM "lesson_completions" WHERE "lesson_completions"."student_id" = $1 AND "lesson_completions"."lesson_id" IN (36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57)  [["student_id", 1]]
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]   Rendered shared/_course_badge.html.erb (0.2ms)
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]    (0.5ms)  SELECT COUNT(*) FROM "lessons" INNER JOIN "sections" ON "lessons"."section_id" = "sections"."id" WHERE "sections"."course_id" = $1  [["course_id", 2]]
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]   Rendered courses/_course_card.html.erb (4.2ms)
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]    (0.5ms)  SELECT "lessons".id FROM "lessons" INNER JOIN "sections" ON "lessons"."section_id" = "sections"."id" WHERE "sections"."course_id" = $1 ORDER BY "lessons"."position" ASC, "sections"."position" ASC  [["course_id", 3]]
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]    (0.4ms)  SELECT COUNT(*) FROM "lesson_completions" WHERE "lesson_completions"."student_id" = $1 AND "lesson_completions"."lesson_id" IN (58, 59, 60)  [["student_id", 1]]
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]   Rendered shared/_course_badge.html.erb (0.1ms)
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]    (0.5ms)  SELECT COUNT(*) FROM "lessons" INNER JOIN "sections" ON "lessons"."section_id" = "sections"."id" WHERE "sections"."course_id" = $1  [["course_id", 3]]
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]   Rendered courses/_course_card.html.erb (3.8ms)
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]    (0.7ms)  SELECT "lessons".id FROM "lessons" INNER JOIN "sections" ON "lessons"."section_id" = "sections"."id" WHERE "sections"."course_id" = $1 ORDER BY "lessons"."position" ASC, "sections"."position" ASC  [["course_id", 4]]
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]    (0.5ms)  SELECT COUNT(*) FROM "lesson_completions" WHERE "lesson_completions"."student_id" = $1 AND "lesson_completions"."lesson_id" IN (61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93)  [["student_id", 1]]
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]   Rendered shared/_course_badge.html.erb (0.1ms)
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]    (0.6ms)  SELECT COUNT(*) FROM "lessons" INNER JOIN "sections" ON "lessons"."section_id" = "sections"."id" WHERE "sections"."course_id" = $1  [["course_id", 4]]
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]   Rendered courses/_course_card.html.erb (6.6ms)
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]    (0.6ms)  SELECT "lessons".id FROM "lessons" INNER JOIN "sections" ON "lessons"."section_id" = "sections"."id" WHERE "sections"."course_id" = $1 ORDER BY "lessons"."position" ASC, "sections"."position" ASC  [["course_id", 5]]
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]    (0.4ms)  SELECT COUNT(*) FROM "lesson_completions" WHERE "lesson_completions"."student_id" = $1 AND "lesson_completions"."lesson_id" IN (94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122)  [["student_id", 1]]
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]   Rendered shared/_course_badge.html.erb (0.5ms)
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]    (0.7ms)  SELECT COUNT(*) FROM "lessons" INNER JOIN "sections" ON "lessons"."section_id" = "sections"."id" WHERE "sections"."course_id" = $1  [["course_id", 5]]
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]   Rendered courses/_course_card.html.erb (6.8ms)
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]    (1.9ms)  SELECT "lessons".id FROM "lessons" INNER JOIN "sections" ON "lessons"."section_id" = "sections"."id" WHERE "sections"."course_id" = $1 ORDER BY "lessons"."position" ASC, "sections"."position" ASC  [["course_id", 6]]
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]    (0.8ms)  SELECT COUNT(*) FROM "lesson_completions" WHERE "lesson_completions"."student_id" = $1 AND "lesson_completions"."lesson_id" IN (123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151)  [["student_id", 1]]
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]   Rendered shared/_course_badge.html.erb (0.2ms)
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]    (0.8ms)  SELECT COUNT(*) FROM "lessons" INNER JOIN "sections" ON "lessons"."section_id" = "sections"."id" WHERE "sections"."course_id" = $1  [["course_id", 6]]
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]   Rendered courses/_course_card.html.erb (10.7ms)
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]    (0.7ms)  SELECT "lessons".id FROM "lessons" INNER JOIN "sections" ON "lessons"."section_id" = "sections"."id" WHERE "sections"."course_id" = $1 ORDER BY "lessons"."position" ASC, "sections"."position" ASC  [["course_id", 7]]
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]    (0.4ms)  SELECT COUNT(*) FROM "lesson_completions" WHERE "lesson_completions"."student_id" = $1 AND "lesson_completions"."lesson_id" IN (152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165)  [["student_id", 1]]
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]   Rendered shared/_course_badge.html.erb (0.2ms)
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]    (0.6ms)  SELECT COUNT(*) FROM "lessons" INNER JOIN "sections" ON "lessons"."section_id" = "sections"."id" WHERE "sections"."course_id" = $1  [["course_id", 7]]
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]   Rendered courses/_course_card.html.erb (12.4ms)
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]   Rendered courses/index.html.erb within layouts/application (188.0ms)
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]   Rendered shared/_head.html.erb (1.3ms)
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]   Rendered shared/_logo.html.erb (0.7ms)
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]   Rendered shared/_navbar_modal.html.erb (3.6ms)
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]   Rendered shared/_logo.html.erb (0.5ms)
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]   Rendered shared/_user_dropdown.html.erb (1.3ms)
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]   Rendered shared/_navbar.html.erb (10.0ms)
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]   AdminFlash Load (1.2ms)  SELECT "admin_flashes".* FROM "admin_flashes" WHERE (expires >= '2018-04-14 04:30:07.958909') ORDER BY created_at desc
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]   Rendered shared/_admin_flash.html.erb (4.7ms)
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]   Rendered shared/_navbar_and_flashes.html.erb (18.4ms)
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]   Rendered shared/_logo.html.erb (0.3ms)
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]   Rendered shared/_footer.html.erb (2.0ms)
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]   Rendered shared/_sidecar.html.erb (0.3ms)
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]   Rendered shared/_ga.html.erb (0.2ms)
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]   Rendered layouts/application.html.erb (217.4ms)
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8] Completed 200 OK in 304ms (Views: 200.5ms | ActiveRecord: 44.9ms)

As you can see, to render each of the courses, three queries are needed:

[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]    (1.1ms)  SELECT "lessons".id FROM "lessons" INNER JOIN "sections" ON "lessons"."section_id" = "sections"."id" WHERE "sections"."course_id" = $1 ORDER BY "lessons"."position" ASC, "sections"."position" ASC  [["course_id", 1]]
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]    (0.7ms)  SELECT COUNT(*) FROM "lesson_completions" WHERE "lesson_completions"."student_id" = $1 AND "lesson_completions"."lesson_id" IN (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35)  [["student_id", 1]]
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]   Rendered shared/_course_badge.html.erb (0.7ms)
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]    (0.9ms)  SELECT COUNT(*) FROM "lessons" INNER JOIN "sections" ON "lessons"."section_id" = "sections"."id" WHERE "sections"."course_id" = $1  [["course_id", 1]]
[9ba5c805-806c-4ff5-9d5e-48fa283a2be8]   Rendered courses/_course_card.html.erb (99.0ms)

The first two are necessary to render the progress badge, as you need to know the number of completed lessons and the total number of lessons for a given course to calculate the percentage. (By the way, the first query is the complex JOIN that could be eliminated by denormalization.)

However, the third query should not be necessary: it is used to render the count (e.g. 53 lessons) here. This is unnecessary because we already fetched all the lesson_ids for the course in the first query, so we can just count that result.

In my opinion, this is a bug/missing feature in Rails, so I opened rails/rails#32569. Assuming the rest of the team agrees with me, it would make a pretty good first PR if someone is interested in contributing to Rails.

Optimize `CoursesController#index`
Currently, a significant amount of time is spent in loading the lesson
objects. This is because they have a big `content` field, so loading
them is very expensive:

1. The database has to read a lot into memory
2. The database has to send back a big result
3. The Ruby driver has to parse/process the big payload
4. The `content` field need to be loaded/copied into big Ruby strings
5. Having these big strings in memory increases memory usage/GC pressure

Therefore, it is a good idea to avoid loading the lesson objects unless
they are absolutely needed to display the content, which is probably
only needed in `LessonsController#show`.

This patch demonstrates a way to refactor the relevant code to avoid
needing to load the lessons object. The focus here is
`CoursesController#index`, but a lot of other endpoints can benefit
from this optimization (you can find them on Skylight).
if user_signed_in?
@user = User.includes(:lesson_completions).find(current_user.id)
end
@user = current_user

This comment has been minimized.

@chancancode

chancancode Apr 14, 2018

Contributor

I left this for now backwards compatibility, since a lot of existing code relies on it. However, as I mentioned in the PR, I suggest that someone else take a pass to remove this from the code base, now that it is (I believe) no longer necessary.

@@ -16,6 +16,11 @@ class User < ApplicationRecord
has_many :projects, dependent: :destroy
has_many :user_providers, dependent: :destroy
def progress_for(course)
@progress ||= Hash.new { |h, c| h[c] = CourseProgress.new(c, self) }

This comment has been minimized.

@chancancode

chancancode Apr 14, 2018

Contributor

It is quite common that we need to get the CourseProgress object for the same user/course more than once in the same request, so this caches it.

@@ -13,8 +13,7 @@
describe 'GET index' do
before do
allow(Course).to receive_message_chain(:order, :includes).
and_return(courses)
allow(Course).to receive(:order).and_return(courses)
end

This comment has been minimized.

@chancancode

chancancode Apr 14, 2018

Contributor

IMO, this test is testing too much of the internal implementation details.

@KevinMulhern

This comment has been minimized.

Member

KevinMulhern commented Apr 14, 2018

Thanks for this brilliant PR @chancancode and thank you for all the detail you provided about the decisions you made. It seems obvious now that eager loading was the problem on these end points after reading your post, but it wasn't something we even considered being the root of the problem 馃檲

The the course progress logic was a part of the code base I always thought was too complex. I'm really impressed with how you've simplified it.

Thanks for your follow up suggestions also, I'll make issues for those straight away so they can be addressed.

In regards to denormalizing the lessons table. We have a project we want to start soon that will require significant changes to our database design. Essentially we want to introduce tracks like Frontend, Backend and Full Stack.

That will require the ability to make custom courses and to change the relationship between sections and courses. A section will need to belong to many courses in the new design.

Database design isn't a strong part of the current maintainer teams collective skillet at the moment. Our best idea at the moment is to create join tables which will probably be very expensive performance wise.

I was wondering if you, @gitKrystan or @zvkemp would be interested in collaborating with us on the new design? we'd love to have input from you guys on it 馃槃

Thanks again for all the work you've done on this, 馃憤 LGTM

@KevinMulhern KevinMulhern merged commit 5b44df1 into TheOdinProject:master Apr 14, 2018

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

@chancancode chancancode deleted the chancancode:optimize-courses-index branch Apr 14, 2018

KevinMulhern added a commit that referenced this pull request Apr 14, 2018

Use Current User instead of Finding the User in Courses Controller
Thanks to great work done this PR #890
we no longer need to find the current user and assign it to an instance variable set the courses controller. We can
instead use current_user directly and avoid performing a redundant query to find the user.
@chancancode

This comment has been minimized.

Contributor

chancancode commented Apr 14, 2018

@KevinMulhern thank you for merging the PR! I am glad you find it useful 馃槃

It seems obvious now that eager loading was the problem on these end points after reading your post

It was not obvious to me at all before digging into it, we actually tried a few things before realizing it 馃槃

I just looked at the deploy on Skylight (I assume you deployed around half an hour ago 鈥 by the way, do you need help setting up deploy tracking on Skylight?). It is probably too early to tell for sure, but it looks like typical is about a wash (or potentially a tiny bit worse), but problem responses likely have improved.

We should probably wait a bit longer before drawing conclusions (since, as we know, the logged in vs guest cases have quite different performance characteristics so a different mix of logged in/guest requests could affect the result). But, honestly I am a little surprised! (I was expecting it to do a little better than that.) It goes to show why you should always check your work on production after deploying, since a lot of variables could be different :P

It looks like, we basically traded one expensive query (fetching and loading lessons 鈥 which is easy from the query planning/execution perspective, but expensive in terms of size) for another:

skylight-6

A few things to note:

  • This is actually the time of 7 queries combined (the last of the three queries in my comment above 鈥 from course.lessons.size), once per each course
  • Also worth pointing out, the time (and allocation) on Skylight is not the raw database time, it times the end-to-end work, including time spent in Ruby (building the query inside Active Record/Arel, etc)
  • I'm not exactly sure why this would be so much slower than the first query (the one returning all the lesson IDs for a given course), as they are essentially doing the same thing. In fact, the first query should be doing slightly more work. As @zvkemp pointed out, perhaps the database is configured differently, or perhaps the difference is on the Ruby side.
  • This was in fact the query that is unnecessary, at least for the case where we already loaded the IDs to show the progress (i.e. for logged in users). I was thinking we would just wait for Rails to fix it (rails/rails#32569) since it didn't seem like a big deal when I looked at it locally. However, the Rails fix probably won't land until Rails 6, and maybe we have more incentive to work around it now until Rails can fix it for us.
  • As Skylight pointed out, it is a "repeated query" of sort, as we are doing 7 of them (once per course). I think there is technically a way to get a similar result in a single query (do a COUNT with a GROUP BY), but it would probably require some clever refactor to find a good spot to do it.
@chancancode

This comment has been minimized.

Contributor

chancancode commented Apr 14, 2018

I was wondering if you, @gitKrystan or @zvkemp would be interested in collaborating with us on the new design? we'd love to have input from you guys on it 馃槃

Of course! Happy to help where we can. Should probably be a new thread at this point :P

KevinMulhern added a commit that referenced this pull request Apr 14, 2018

Use Current User instead of Finding the User in Courses Controller (#892
)

Thanks to great work done this PR #890
we no longer need to find the current user and assign it to an instance variable set the courses controller. We can
instead use current_user directly and avoid performing a redundant query to find the user.
@KevinMulhern

This comment has been minimized.

Member

KevinMulhern commented Apr 14, 2018

Thanks @chancancode, yeah I deployed around that time. I wasn't aware of deploy tracking on Skylight. I've just enabled it 馃槃

It seems to have comparable performance to the previous query, which isn't a problem since we cleaned up and simplified a crufty part of the code base significantly with no performance losses 馃帀

We could look into caching some of the lesson and course queries as they don't change all that often or moving some of these things into query objects.

Thanks for being willing to help us on these db issues and all the great work you've done here 馃槃

@chancancode

This comment has been minimized.

Contributor

chancancode commented Apr 27, 2018

Upstream Rails issue is fixed (for Rails 6)!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment