Explore ways to reload gems

[fxn]

(Discussion started in fxn/zeitwerk#287.)

The purpose of this ticket would be to explore ways to let Rails reload some gems. This could be useful if the gem is being actively developed.

The documentation of Zeitwerk warns this may be tricky:

In a given process, ideally, there should be at most one loader with reloading enabled. Technically, you can have more, but it may get tricky if one refers to constants managed by the other one. Do that only if you know what you are doing.

In the proposed use case, the application refers to constants in the gem, and the gem does not refer to constants from the application. This is important.

/cc @matthewd

[rafaelfranca]

Any reason why this doesn't work today? We have many gems in our application (70+) and they are all reloaded. They just need to be Rails engines and inside the application directory.

[fxn]

@rafaelfranca I suggested this, but @davidmilo does not seem to want its gem to be an engine.

@davidmilo to help discuss, could you please be more explicit about your use case?

[rafaelfranca]

That is the contract that Rails expects. If a library needs to integrate well with Rails and its reloaders it needs to include a Engine. I don't think we should change that contract.

[davidmilo]

I am a huge fan of extracting code from rails apps into gems. With larger codebases there are many things which can be extracted and either shipped as a.) public gem for everyone to use or b.) private gem which can be used in other product you are using within the company. I think that moving logic to gems helps you modularise large rails app better. It makes it easier to re-use lot of code across different products.

These gems are, in many cases, not related to rails at all. They either extract some piece of business domain specific to the company or product you are working on. Sometimes they can be some nice algorithm solving general problem. Some examples could:

• gem which knows how to talk to some API service we are using which maybe doesn't have ruby client.
• gem which knows how to talk to some internal service we are using in the company - for example wrapper on how to talk to our specific instance of Solr/Elastic search/
• gem made to crawl specific websites (I worked at a place which was crawling 50 public websites for different sources of information) - organising these different crawlers into smaller gems which know how to handle specific sources can be quite useful.
• gem to wrap up our home made XML diffing algorithm
• gem which wraps our internal design system

Why should these be Rails engines? They have nothing in common with Rails - they are not additions or any extensions of Rails. They do not add any features to the framework. They are just pure ruby classes doing something which can be used anywhere. Why should I make them dependant on Rails?

I could think about 3 different cases when extracting into gems would come handy:

Example 1

• We have a huge product which is the main product of company A. Company wants to make some new products where lot of logic/tooling could be shared.
• My first step would to make local gems along side my main product. This might take some time because code extraction can sometimes be difficult. You need to find abstraction of the problem. It can be sometimes tricky to extract logic from Rails if code was not made with good separation of concerns and when code is too bound to the framework.

my_repo/
  my_rails_app/
  my_gem_1/
  my_gem_2/
  my_gem_3/

• Main product is the main consumer of these gems. Majority of changes and features are developed along side the development of the product. We can easily modify code of the gems and code of the product at the same time. These gems can be published to a private company repository as needed when it is meaningful to do a new version release.
• We can eventually start making new products which can start re-using these gems to re-use lot of code which would be similar across the products.

Example 2
Mono repos are getting extremely popular! They come with lot of benefits. It would be very common to run in a setup where you have 3-4 products and 5-7 gems in a same repo

my_mono_repo/
  my_product_1/ # rails app
  my_product_2/ # rails app
  my_product_3/ # rails app
  my_gem_1/
  my_gem_2/
  my_gem_3/
  my_gem_4/
  my_gem_5/

Working across 8 different github repos can create lot of maintenance, manual work, headache and lot of wasted time. Mono repos come with their own problems, but benefits of being able to change some shared code and directly update products using it at the same time can provide huge boosts to development process. Imagine opposite process:
Updating gem in a github repo. Releasing the gem. Then update 3 other repos which are using the gem and release/redeploy them separately. Lot of these things can be easily simplified by mono repo structure and good CI setup.

What does Rails framework want to do about this new boom of mono repos? Should there be some support for it to make it easier to work on several different Rails products and share code in between?

Example 3

• I am making a framework which crawls lot of different public authority websites and collects different legal information/regulations etc into a single place.
• My Rails all is the framework itself which runs different crawlers.
• To keep code nicely separated, it is good idea to make each crawler its own gem. Small gem which knows how to crawl single website and returns what was found in some generalised format. Each gem is easy to re-use, test and develop on it's own. Good separation of concerns - gems doesn't need to know about rest of the Rails app or other crawlers.
• In this case gems will probably never be used by other products but process of extracting code into small gems helps keep codebase more clean and understandable.
• Gems helps you to more clearly define dependencies and creates more natural modules.

[davidmilo]

Maybe I am not understanding the contract you are talking about well. I guess your idea would be that everything which wants to be used in Rails should ship with rails engine? Am I understanding it correctly? I think it is little weird to require that of gems and libraries which does not directly interact or depends on Rails. I think separation that:

• gems which adds direct logic/features to the framework, are Rails engines
• gems which are just pure ruby classes which can be used any other framework or just other pure ruby code, should NOT be Rails engines.

[rafaelfranca]

Why should these be Rails engines?

Because you want them to be reloaded in a Rails application. Engines aren't for telling a gem has to do with Rails. It is to teach Rails applications what to do with those gems.

The gems don't become Rails engines, they include Rails engines. For this gem to be used by other frameworks it doesn't need to load the Rails Engine.

So by including a Rails::Engine subclass in your gem your not making your gem a Rails engine, you are including an optional glue to to your gem for it to be able to be properly glued to a Rails application.

Note that I'm telling you to do rails plugin new my_gem, I'm only telling you to include a subclass of Rails::Engine. Like many gems, that can and are used outside Rails app do.

https://github.com/drapergem/draper/blob/26a18c8cc9ce112f7cf2a308b52952680d9a2cdf/lib/draper.rb#L32
https://github.com/instacart/makara/blob/9e7960558a75aed3f97ba4cbab61abb64687ec3c/lib/makara.rb#L3
https://github.com/thoughtbot/bourbon/blob/71d4776757bbd0d7da47b5660eb6d1f3bf73fc3f/lib/bourbon.rb#L5
https://github.com/carrierwaveuploader/carrierwave/blob/ed8799191824a4d2762eb1028c01220102699377/lib/carrierwave.rb#L51
https://github.com/collectiveidea/delayed_job/blob/b66bb64437a8606a414a390531ac73108911434e/lib/delayed/railtie.rb#L5
https://github.com/primer/octicons/blob/6bc78de32be1a218a69e10a731072331ba94e7f9/lib/octicons_helper/lib/octicons_helper.rb#L3

[fxn]

I believe that approach is not quite right for this use case.

The gem needs its own autoloader, because when it ships it needs to autoload its code by itself. And the gem has no initializer to run or any integration to be done with Rails projects.

Also, when the gem ships, Rails projects using the gem should not reload the gem.

Point is, it would be convenient to reload the gem's loader while the gem is being developed and used via a Rails application.

This does not necessarily mean we have to do anything in Rails, eh? This ticket only wants to trigger a discussion.

I have a busy week and have not been able to sit down and think about this, but my main ideas to be explored are:

• The gem should expose its loader instance.
• The gem should expose its project tree.
• The Rails application should be configured to watch the gem's project tree.
• If the gem's source code changes, when the time arrives in Rails's liefcycle, the gem's loader would be reloaded first, and then the main loader would be reloaded. This order is important.

If something like that works with existing APIs, we do not need to do anything, the outcome of the discussion would be what to put in David's app initializers to make this happen.