New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Experimental code reloading #249

Closed
jodosha opened this Issue Jun 5, 2015 · 17 comments

Comments

Projects
None yet
5 participants
@jodosha
Member

jodosha commented Jun 5, 2015

Overview

Lotus itself doesn't implement code reloading.

We use a reloadable context only for development mode via lotus server command. The reason is simplicity. Once we bypass that reloadable context, Lotus works like any other Ruby library, without messing with constant autoloading tricks.

The library we use for reloading is: Shotgun. It's written in Ruby and uses fork(2) to spawn a new process when a new request comes it. It has portability issues: it works only on MRI (see #237).

Experimental Solution

We want to keep this architecture: Lotus shouldn't care about code reloading. We should have this feature implemented on the outside, still with lotus server.

There is a POSIX compatible solution that provides code reloading for generic processes: entr(1).
I did some manual tests and seems to work well. The idea is to replace Shotgun with Entr in the long term. Before to do that, we should check if it works properly for Lotus.

Pros

  • It removes a Rubygem runtime dependency for lotusrb
  • It's probably faster, as it's written in C, instead of Ruby
  • It can improve portability with other Ruby implementations

Cons

  • It requires dev machines to install entr. Note that it has both apt-get and brew installers.

Details

We need to add a CLI arg for lotus server: --experimental-reloading.
When used, it should prefer entr over shotgun.
If the binary can't be found, it should raise an exception.

The internal implementation should resemble this command:

find . | entr -r ...

http://entrproject.org/

@jodosha jodosha added the enhancement label Jun 5, 2015

@jodosha jodosha self-assigned this Jun 5, 2015

@jodosha jodosha modified the milestone: v0.4.0 Jun 5, 2015

@pascalbetz

This comment has been minimized.

Contributor

pascalbetz commented Oct 20, 2015

@jodosha
What needs to be done for entr to work? Can you try the streaming sample with entr (if you think that it might be shotgun which prevents streaming from working) or explain how you used entr.

Details:
#303

@thecatwasnot

This comment has been minimized.

thecatwasnot commented Oct 21, 2015

@pascalbetz I played with using entr on a lotus project inside of docker, basically running entr on host machine, to restart containers. Dropped it in a Makefile:

.PHONY: watch
watch:
        docker-compose kill dev
        while sleep 1; do find apps/ config/ lib/ config.ru Gemfile.lock ! -name '*.sw*' | entr -dr docker-compose up dev; done

.PHONY: watch-bundler
watch-bundler:
        ls Gemfile | entr docker-compose run dev bundle install

.PHONY: watch-spec
watch-spec:
        while sleep 1; do find apps/ lib/ spec/ ! -name '*.sw*' | entr -d docker-compose run dev bundle exec rspec; done

The main container comes up with bundle exec rackup -p 3000 --host 0.0.0.0 (you shouldn't need the host flag if you're not using docker.)
I remember I did need a new version of entr, build instructions @ https://bitbucket.org/eradman/entr/ and it did take a fair bit of fiddling to get flags etc just right. Hopefully this can help, even if it doesn't make it into lotus proper.

@pascalbetz

This comment has been minimized.

Contributor

pascalbetz commented Oct 21, 2015

@thecatwasnot thanks!

@jodosha

This comment has been minimized.

Member

jodosha commented Oct 21, 2015

@pascalbetz are you looking into this as well? 😉

@pascalbetz

This comment has been minimized.

Contributor

pascalbetz commented Oct 21, 2015

@jodosha
Not really. I just wanted to see if entr works with streaming because shotgun breaks streaming because the way it's implemented. Since entr goes a somewhat different way i was hoping that it works with streaming. If you can show me how you used it, then i can try it out.

@jodosha

This comment has been minimized.

Member

jodosha commented Oct 21, 2015

@pascalbetz Okay, I just played with entr months ago and looked promising. Nothing more.

history | grep entr
# ...
 1867  which entr
 1868  ls * | entr -r bundle exec lotus server --no-code-reloading
 1869  ls *.rb | entr -r bundle exec lotus server --no-code-reloading
 1870  ls -R * | entr -r bundle exec lotus server --no-code-reloading
 1872  ls -R * | entr rackup
 1873  ls -R * | entr -r rackup
 1874  find . | entr -r rackup
 1875  find . | entr -r bundle exec lotus server --no-code-reloading
# ...

We can accept the incompatibility between shotgun and streaming as a known limitation for now. But yeah, that is another case that supports the need of switch tool like suggested by this ticket.

@Eein

This comment has been minimized.

Eein commented Nov 24, 2015

Isn't fork() and shotgun one of the only major dependency issues with windows compatibility?

I personally don't use windows for development, but if there's an option that is cross platform without losing threading or streaming; It might be kind to consider those who use windows if you're at a crossroad on this issue.

Just bringing light to this - as it doesn't affect me :)

@jodosha

This comment has been minimized.

Member

jodosha commented Nov 24, 2015

@Eein Hey, thanks for joining this conversation. It's true, Windows cross compatibility would be great to achieve. Do you know a tool that offers the same features of entr? Thank you very much.

@Eein

This comment has been minimized.

Eein commented Nov 27, 2015

I looked for another solution that had windows support but did not find anything. I'm actually wondering if entr will compile on windows with a custom makefile - it doesn't make any mention but once i get on a suitable windows machine i'll try and give it a shot. If you do decide to use entr in lotus i'll give it a test run on windows and report any issues

@Eein

This comment has been minimized.

Eein commented Jan 27, 2016

Just wanted to update - I tried toying with this a couple of months ago to no avail. entr just wont work with windows due to kqueue/inotify depedencies, and replacing those dependencies with case-by-case OS alternatives proved to be a little too difficult for me and would cause a huge dependency chain that I dont really think this project needs. :( Without extensive experience with code-reloading, I would likely be creating software that wouldn't work as intended for this use case. :(

One solution i was looking at was situationally replacing inotify with https://github.com/thekid/inotify-win in a seperate build target for windows, but could not get it working properly. by swapping this dependency, you'd also have to try using fswatch on mac instead.

The last option that i haven't tried yet was building some tooling on top of libuv which might be the best option - theres some good ruby bindings that support filesystem events as well which might make it easy to implement.

Also i think its thread safe... may be wrong though.

Edit: libuv is apparently not thread-safe, not sure if thats an issue for this task.

@jodosha

This comment has been minimized.

Member

jodosha commented Feb 9, 2016

@Eein Thanks again for the detailed explanation.

Because this is a development tool, thread safety isn't an issue.

@jodosha

This comment has been minimized.

Member

jodosha commented Feb 9, 2016

At this point my suggestion is to proceed like this:

  1. Remove shotgun as hard dependency from hanami.gemspec
  2. Generate new projects with this gem in Gemfile (see below)
  3. For the server: detect if Shotgun is present and activate code reloading via Shotgun (which is the actual implementation)
  4. Otherwise try to fallback to entr
  5. When none of them isn't available, it should not try to activate code reloading.

Gemfile for new projects:

group :development do
  # Shotgun provides code reloading support for `hanami server`.
  # If you are on a POSIX system, please use `entr`.
  # If not possible, please use Shotgun.
  # Read more at: [LINK TO HANAMI GUIDES]
  # gem 'shotgun'
end

@hanami/contributors WDYT?

@jodosha

This comment has been minimized.

Member

jodosha commented Feb 9, 2016

Case in favor of this change: NewRelic doesn't support Shotgun.

@Eein

This comment has been minimized.

Eein commented Feb 9, 2016

I think thats fair - if someone wants to implement an additional windows fallback, it doesn't seem like it would be too hard with your proposal.

@jodosha jodosha assigned aderyabin and unassigned jodosha Feb 16, 2016

@aderyabin

This comment has been minimized.

Contributor

aderyabin commented Feb 17, 2016

@jodosha. As I understand, entr watching files and restarts application. Why do we not use rerun or guard?

@jodosha

This comment has been minimized.

Member

jodosha commented Feb 18, 2016

@aderyabin Gem dependencies.

Using rerun means 6 runtime dependencies for Hanami:

GEM
  remote: https://rubygems.org/
  specs:
    ffi (1.9.10)
    listen (3.0.6)
      rb-fsevent (>= 0.9.3)
      rb-inotify (>= 0.9.7)
    rb-fsevent (0.9.7)
    rb-inotify (0.9.7)
      ffi (>= 0.5.0)
    rerun (0.11.0)
      listen (~> 3.0)

PLATFORMS
  ruby

DEPENDENCIES
  rerun

BUNDLED WITH
   1.11.2

While guard means 16 runtime gems:

GEM
  remote: https://rubygems.org/
  specs:
    coderay (1.1.0)
    ffi (1.9.10)
    formatador (0.2.5)
    guard (2.13.0)
      formatador (>= 0.2.4)
      listen (>= 2.7, <= 4.0)
      lumberjack (~> 1.0)
      nenv (~> 0.1)
      notiffany (~> 0.0)
      pry (>= 0.9.12)
      shellany (~> 0.0)
      thor (>= 0.18.1)
    listen (3.0.6)
      rb-fsevent (>= 0.9.3)
      rb-inotify (>= 0.9.7)
    lumberjack (1.0.10)
    method_source (0.8.2)
    nenv (0.3.0)
    notiffany (0.0.8)
      nenv (~> 0.1)
      shellany (~> 0.0)
    pry (0.10.3)
      coderay (~> 1.1.0)
      method_source (~> 0.8.1)
      slop (~> 3.4)
    rb-fsevent (0.9.7)
    rb-inotify (0.9.7)
      ffi (>= 0.5.0)
    shellany (0.0.1)
    slop (3.6.0)
    thor (0.19.1)

PLATFORMS
  ruby

DEPENDENCIES
  guard

BUNDLED WITH
   1.11.2

@jodosha jodosha modified the milestone: v0.8.0 Feb 26, 2016

@jodosha

This comment has been minimized.

Member

jodosha commented Mar 10, 2016

Implemented by #523

@jodosha jodosha closed this Mar 10, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment