Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker debugging and sub-debugger with random port #107

Open
jcavalieri opened this issue Jul 20, 2017 · 38 comments

Comments

@jcavalieri
Copy link

commented Jul 20, 2017

Hi,

I'm trying to debug rails on a docker container using RubyMine. For lack of a better term, I do have an initial successful hand shake. However, the debugger then creates a "sub-debugger" on a different port. This port seems to be random, therefore, I can't pre-configure to have that port shared. Since the IDE doesn't respond to the sub-debugger port being open, then my rails site doesn't start.

root@f79144807c32:/myapp# rdebug-ide -d --host 0.0.0.0 --port 1234 --dispatcher-port 26162 -- /usr/local/bundle/bin/rails s -b 0.0.0.0 -p 3000 -e development
Fast Debugger (ruby-debug-ide 0.6.1.beta4, debase 0.2.2.beta9, file filtering is supported) listens on 0.0.0.0:1234
Connected from 10.211.55.2
5: Starting control thread
5: Processing in control: b /myapp/app/api/v1/users.rb:6
5: <breakpointAdded no="1" location="/myapp/app/api/v1/users.rb:6"/>
5: Processing in control: start
5: Starting: running program script
Fast Debugger (ruby-debug-ide 0.6.1.beta4, debase 0.2.2.beta9, file filtering is supported) listens on 0.0.0.0:37922
5: Ide process dispatcher notified about sub-debugger which listens on 37922

Note the port 37922. I'm having trouble finding documentation on the sub-debugger. Do you have any suggestions on how to get around this?

Other info, I'm not using "Docker for Mac" I installed docker tools via brew and and my default docker machine is using the parallels driver.

-John

@jcavalieri

This comment has been minimized.

Copy link
Author

commented Jul 20, 2017

Looks like this is a duplicate (or related):
#73

I'm a bit confused how some folks on the web say they got this working (without SSH).

@ivanprado

This comment has been minimized.

Copy link

commented Jan 4, 2018

Hi @jcavalieri . Did you found any solution to that? I'm stuck on the same problem.

@jcavalieri

This comment has been minimized.

Copy link
Author

commented Jan 4, 2018

Hi @ivanprado,
I haven't. However, rubymine does claim that they support it now. I was about to give that a try.
-John

@tatsuco

This comment has been minimized.

Copy link

commented Jan 30, 2018

I have the same problem.
Is the problem solved? If the problem is solved, I would like to know how to do it.

@jcavalieri

This comment has been minimized.

Copy link
Author

commented Jan 30, 2018

I haven't solved it. RubyMine states to have a solution but I couldn't get it to work. I think what RubyMine is doing is adding their own container one's docker environment.

I haven't heard from the maintainers.

@valich, I see you are the most active recently on this repo. Any thoughts on this?

-John

@valich

This comment has been minimized.

Copy link
Contributor

commented Jan 31, 2018

We'll recheck how the workaround with using docker container as SSH remote interpreter work (have you tried it?). Generally debugging multiprocess stuff in Docker may be cumbersome indeed due to automatic port generation for subprocesses. We are planning to do some refactoring to use single port for the whole debug session, but I doubt we'll make it to 2018.1.

Right now I suppose one should either try to set up SSH connection, or we can make some fork with workarounds for specifying fixed ports.

@valich

This comment has been minimized.

Copy link
Contributor

commented Jan 31, 2018

OK we discussed and found quite a simple solution of moving to a single-port scheme without much refactoring (at least on IDE side) so we'll try to implement it.

In short, debuggee sets up a proxy to communicate with IDE via multiple connections (opposed to multiple connections from different debuggee processes). Subprocesses will communicate with the proxy with the request to forward the data to the outer client (IDE). The mapping (inner process <-> outer connection) will be also used to deliver responses (commands) back to subprocesses.

@jcavalieri

This comment has been minimized.

Copy link
Author

commented Jan 31, 2018

That's great to hear @valich!

@noizex

This comment has been minimized.

Copy link

commented Feb 20, 2018

@valich, that sounds great! Any update on this? I used patch for rdebug-ide where I could provide fixed set of ports that were opened for subprocesses, but it really is PITA - it requires opening these ports and adds complexity. Running this in some proxy mode sounds like a good idea - will it work for existing IDEs without requiring any changes? Also any idea when some initial version could be expected? Happy to give it a try.

@valich

This comment has been minimized.

Copy link
Contributor

commented Feb 22, 2018

@noizex I think it just entered the state "in progress". I think it should be ready to try in a couple of weeks, no earlier.

@aaronblenkush

This comment has been minimized.

Copy link

commented Mar 1, 2018

I was able to get this working over an ssh connection to a remote server (I have not tried this on a Docker container) by doing the following:

  • [Terminal 1]
    • ssh -L 1234:localhost:1234 -R 26162:localhost:26162 $remoteHost
    • rdebug-ide --debug --host 0.0.0.0 --port 1234 --dispatcher-port 26162 -- bin/rails s thin
  • [RubyMine]
    • Attach the debugger.
  • [Terminal 1]
    • Take note of the sub-debugger port ($subPort)
  • [Terminal 2]
    • ssh -L $subPort:localhost:$subPort $remoteHost

Note: At first I tried using bundle exec thin start... as the rdebug-ide command, but it seemed to create two subdebuggers, and the thin server never started.

@andrejska

This comment has been minimized.

Copy link

commented Jun 20, 2018

@valich any update on this ticket?

@noizex

This comment has been minimized.

Copy link

commented Jun 20, 2018

Also very interested in this, could help with some testing if needed, or anything else - just shout :)

@jcavalieri

This comment has been minimized.

Copy link
Author

commented Sep 11, 2018

Hi @aaronblenkush ,
We want to avoid having to set up an SSH daemon on every docker container.

Hi @valich, did the proposed solution get any movement?

@vceron

This comment has been minimized.

Copy link

commented Sep 19, 2018

I'm interested in a solution too. I've the same problem and can't find a solution otherwise using single cluster on Puma configuration.

@mrbiggred

This comment has been minimized.

Copy link

commented Dec 9, 2018

To work around this issue I overrode the find_free_port method in the lib/rubyd-debug-ide/multiprocess/pre_child.rb file and opened the port in my Docker container. I like this work around because I don't have to setup SSH in my Docker container.

# initializers/ruby-debug-ide.rb

# In non-dev and rake tasks the Ruby Debug IDE gem
# is not loaded so nothing to override.
if defined?(Debugger::MultiProcess)
  Debugger::MultiProcess.module_eval do
    class << self
      def find_free_port(host)
        server = TCPServer.open(host, 58438)
        port   = server.addr[1]
        server.close
        port
      end
    end
  end
end

In my Docker-Compose file I added the port:

web:
    ports:
      - "58438:58438"
      <Other ports, such as 8080 for Unicorn>

Now the debugger no longer hangs when spawning the second process. It used to hang after the second "Fast Debugger" line but now it works as expected.

Fast Debugger (ruby-debug-ide 0.6.1, ruby-debug-base19x 0.11.30.pre15, file filtering is not supported) listens on 0.0.0.0:1234
web_1       | => Booting Unicorn
web_1       | => Rails 3.2.16 application starting in development on http://0.0.0.0:8080
web_1       | => Call with -d to detach
web_1       | => Ctrl-C to shutdown server
web_1       | listening on addr=0.0.0.0:8080 fd=13
web_1       | worker=0 spawning...
web_1       | master process ready
web_1       | Fast Debugger (ruby-debug-ide 0.6.1, ruby-debug-base19x 0.11.30.pre15, file filtering is not supported) listens on 0.0.0.0:58438
web_1       | worker=0 spawned pid=10
web_1       | worker=0 ready
web_1       | 
web_1       | 
web_1       | Started GET "/" for 172.19.0.1 at 2018-12-09 15:48:36 +0000
                   <Never used to get to the above GET command>

In development I only use one process so I just hard coded the port but you could setup a range if you wanted too.

Details:

I'm supporting an older application that previously used Thin as it's development webserver but Unicorn in production. Part of the application is multi-threaded which would hang the Thin webserver since it's single threaded by default. I decided to use Unicorn in development to be consistent with production and when I did the switch I ran into the above issue when debugging. There was no issues when running the application using Unicorn without debugging (i.e. clicking the green arrow).

Ubuntu 18.04 LTS
RubyMine 2018.3
Ruby 1.9.3
Rails 3.2
Ruby Debug IDE Gem 1.6.1
Ruby Debug Base 19x 0.11.30-pre15
Unicon 4.3.1

Note : The find_free_port method is moved in the 0.7 beta of Ruby Debug IDE to lib/ruby-debug-ide/ruby-debug-ide.rb.

Feedback is welcome,.

Edit (Dec 11/18): Added check that the Debugger::MultiProcess is defined.

@vmassuchetto

This comment has been minimized.

Copy link

commented Dec 21, 2018

@mrbiggred how is this supposed to work if the initializers are loaded only when rdebug-ide launches the $COMMAND$ -- therefore, after the dispatcher hangs listening on the random port?

If anyone can help me, this is the ouput for a ruby:2.3-alpine3.7 Docker container:

bash-4.4# bundle exec rdebug-ide --debug --host 0.0.0.0 --port 1234 --dispatcher-port 26162 -- bin/rails server
Fast Debugger (ruby-debug-ide 0.6.1, debase 0.2.2, file filtering is supported) listens on 0.0.0.0:1234
Connected from 172.18.0.1
1324: Starting control thread
1324: Processing in control: catch off
1324: Processing in control: start
1324: Starting: running program script
1324: Processing in control: thread listFast Debugger (ruby-debug-ide 0.6.1, debase 0.2.2, file filtering is supported) listens on 0.0.0.0:33141
1324: Ide process dispatcher notified about sub-debugger which listens on 33141
@mrbiggred

This comment has been minimized.

Copy link

commented Dec 21, 2018

@vmassuchetto if I understand your question correctly you are asking why does overriding the find_free_port method work. It works because instead instead or returning a random port when spawning a new process a known port is returned. That known port can be opened on the Docker container so the RubyMine, running on your host, can connect and debug the spawned process.

As for your ruby:2.3-alpine3.7 output it appears RubyMine can't connect to port 33141. Did you open that port on your Docker container?

If I misunderstood your questions please let me know. Finally, it is possible my workaround has a bug and/or only works for my particular setup.

@vmassuchetto

This comment has been minimized.

Copy link

commented Dec 22, 2018

@mrbiggred Sorry if I was not clear. I'm trying to attach RubyMine to a running rdebug-ide instance. My problem seems to be exactly the same as the OP.

I just copied the find_free_port method. So I would expect the Docker port to be 58438, which continues to be a random port.

As you can see in our output, rails didn't get called, so initializers/ruby-debug-ide.rb can't be loaded at all. What's the correct configuration on RubyMine for this case?

@vmassuchetto

This comment has been minimized.

Copy link

commented Jan 24, 2019

Got it working by using a rails launch script without bundler (load Gem.bin_path("railties", "rails") call).

@jcavalieri

This comment has been minimized.

Copy link
Author

commented Jan 30, 2019

I think this wouldn't be too difficult to add a new command line option like --open-port-range. I'll give making an PR a try with this. I'm thinking --open-port-range 7000:8000. And this would work with an exposed port range in the dockerfile.

@ViugiNick

This comment has been minimized.

Copy link
Contributor

commented Jan 31, 2019

@jcavalieri @vmassuchetto @mrbiggred There is a prototype of the debugger that uses only one port for all processes. It would be very helpful if you tried it. #164
https://github.com/ViugiNick/ruby-debug-ide/tree/so_reuseport

@jcavalieri

This comment has been minimized.

Copy link
Author

commented Feb 3, 2019

Hi @ViugiNick , I like what you are doing in your branch. Have you created an PR for this? I'm wondering what the maintainers would think of your solution.

@jcavalieri

This comment has been minimized.

Copy link
Author

commented Feb 3, 2019

Ha! @ViugiNick I see that you are a maintainer! I'll try out your branch and give you feedback.

@mrbiggred

This comment has been minimized.

Copy link

commented Feb 12, 2019

@ViugiNick what is the easiest to test your fix? I'm assuming the following updates to my Gemfile would work:

gem 'ruby-debug-ide' git: 'https://github.com/ViugiNick/ruby-debug-ide.git' branch: 'so_reuseport'

Will this replace the existing ruby-debug-ide gem since they will have the same version number? Is it better to manually delete the previous version first? Thank you.

@ViugiNick

This comment has been minimized.

Copy link
Contributor

commented Feb 12, 2019

@mrbiggred Yes, it's better to delete the old one manually

@locofocos

This comment has been minimized.

Copy link

commented Apr 1, 2019

@ViugiNick I tested out your prototype. For my situation, it unfortunately isn't working any better. I'm not sure where I would go to offer any further debugging info about why the connection is failing with your prototype.

I've made this change to my gemfile:

  gem 'ruby-debug-ide', git: 'https://github.com/ViugiNick/ruby-debug-ide.git', branch: 'so_reuseport'

which brought in commit 593eef1 of your branch into my gemfile.lock.

My situation:
I'm using Rubymine, trying to open up a remote debugger to a ruby/rails application running inside docker. When I run the ruby/rails application without docker, I'm able to open a debugging connection using this gem just fine. Using docker gives me this error message:

Failed to find free socker port for process dispatcher

I did some snooping in wireshark without using Docker, to see if there was some port being used other than the ports listed in the commands I'm running. I saw 1 or 2 ephemeral ports being used every time (which I haven't explicitly configured anywhere). That led me #73 and then here.

OS - Mac 10.13
Docker - 2.0.0.0-mac81
Ruby - 2.6
Rubymine - 2018.3.5
ruby-debug-ide - 0.6.1
Rails server - passenger 6.0

My docker-compose file looks like this:

  debug:
    command: sh -c 'rm -rf ./tmp && bundle exec rdebug-ide --host 0.0.0.0 --port 1234 --dispatcher-port 26162 -- bin/rails s -b 0.0.0.0'
    ports:
      - 3000:3000
      - 1234:1234
      - 26162:26162

Thanks for working to fix the issue!

@ViugiNick

This comment has been minimized.

Copy link
Contributor

commented Apr 1, 2019

@locofocos Thanks for the feedback, I'll try to fix it ASAP.

In fact, this prototype still needs two ports (not more)

@ViugiNick

This comment has been minimized.

Copy link
Contributor

commented Apr 3, 2019

@locofocos
OS - Mac 10.13
Docker - 2.0.0.0-mac81
Ruby - 2.6
Rubymine - 2018.3.5
ruby-debug-ide - 0.6.1
Rails server - passenger 6.0

Looks like you are still using the old version of ruby-debug-ide gem

@ViugiNick

This comment has been minimized.

Copy link
Contributor

commented Apr 3, 2019

@jcavalieri @ivanprado @tatsuco @noizex @locofocos I fixed the bug in my branch, so it would be very nice if you could give it a try (https://github.com/ViugiNick/ruby-debug-ide/tree/so_reuseport)

@howtwizer

This comment has been minimized.

Copy link

commented Apr 18, 2019

have same issue Exception: Connection refused - connect(2) for "192.168.16.1" port 26168 as example. @ViugiNick - I had try your fork but it didn't fix it. Thank's for the work

@ViugiNick

This comment has been minimized.

Copy link
Contributor

commented Apr 18, 2019

@howtwizer do you have this port opened?

@howtwizer

This comment has been minimized.

Copy link

commented Apr 18, 2019

@ViugiNick hm. good question.

app:
    build:
      context: ../../
      dockerfile: devops/dev/Dockerfile
    command: "bundle exec rails s -b 0.0.0.0"
    container_name: test_app
    depends_on:
      - redis
      - db
    env_file: .env
    image: "test_app:dev"
    ports:
      - "3000:3000"
      - "26162-26168"
    stdin_open: true
    tty: true
    volumes:
      - "../../:/usr/src/test_app"
      - "bundle:/usr/local/bundle"

Here is my part of docker-composer.yml I had add "26162-26168" but still have error. May be I'm doing it wrong. Do not have a lot of experience with it.

@peter-grainger

This comment has been minimized.

Copy link

commented May 28, 2019

@ViugiNick I have a slightly different issue

Exception: undefined method `chomp' for nil:NilClass
/usr/lib/ruby/gems/2.4.0/bundler/gems/ruby-debug-ide-ab971e6097a4/lib/ruby-debug-ide.rb:169:in `block in notify_dispatcher_if_needed'
/usr/lib/ruby/gems/2.4.0/bundler/gems/ruby-debug-ide-ab971e6097a4/lib/ruby-debug-ide.rb:166:in `times'
/usr/lib/ruby/gems/2.4.0/bundler/gems/ruby-debug-ide-ab971e6097a4/lib/ruby-debug-ide.rb:166:in `notify_dispatcher_if_needed'
/usr/lib/ruby/gems/2.4.0/bundler/gems/ruby-debug-ide-ab971e6097a4/lib/ruby-debug-ide.rb:123:in `block in start_control'
Fatal exception in DebugThread loop: undefined method `accept' for 3:Integer
@noizex

This comment has been minimized.

Copy link

commented May 28, 2019

@peter-grainger seems like your dispatcher is not running, the error indicates that it failed to open connection to a given

acceptor_host, acceptor_port = ENV['IDE_PROCESS_DISPATCHER'].split(":")
acceptor_host, acceptor_port = '127.0.0.1', acceptor_host unless acceptor_port

s.gets.chomp just reads from the socket you open here https://github.com/ViugiNick/ruby-debug-ide/blob/so_reuseport/lib/ruby-debug-ide.rb#L170 and if it's nil it failed to do so.
So make sure your dispatcher is running and listening at provided host & port

@peter-grainger

This comment has been minimized.

Copy link

commented May 28, 2019

@noizex I'm running in a docker container and need to attach to IP 0.0.0.0 where would I set that to take effect?

@noizex

This comment has been minimized.

Copy link

commented May 28, 2019

I guess putting IDE_PROCESS_DISPATCHER=0.0.0.0:1234 (or whatever is your host/dispatcher port) before the command, but I also think ruby-debug-ide allows passing this as command line arguments too.

@peter-grainger

This comment has been minimized.

Copy link

commented May 28, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.