Developer's guide to ngrok
The ngrok project is composed of two components, the ngrok client (ngrok) and the ngrok server (ngrokd). The ngrok client is the more complicated piece because it has UIs for displaying saved requests and responses.
git clone firstname.lastname@example.org:inconshreveable/ngrok.git cd ngrok && make bin/ngrok [LOCAL PORT]
There are Makefile targets for compiling just the client or server.
make client make server
NB: You must compile with Go 1.1+! You must have Mercurial SCM Installed.
Compiling release versions
Both the client and the server contain static asset files. These include TLS/SSL certificates and the html/css/js for the client's web interface. The release versions embed all of this data into the binaries themselves, whereas the debug versions read these files from the filesystem.
You should always develop on debug versions so that you don't have to recompile when testing changes in the static assets.
There are Makefile targets for compiling the client and server for releases:
make release-client make release-server make release-all
The strategy I use for developing on ngrok is to do the following:
Add the following lines to /etc/hosts:
127.0.0.1 ngrok.me 127.0.0.1 test.ngrok.me
Run ngrokd with the following options:
./bin/ngrokd -domain ngrok.me
Create an ngrok configuration file, "debug.yml" with the following contents:
server_addr: ngrok.me:4443 tunnels: test: proto: http: 8080
Then run ngrok with either of these commands:
./bin/ngrok -config=debug.yml -log=ngrok.log start test ./bin/ngrok -config=debug.yml -log=ngrok.log -subdomain=test 8080
This will get you setup with an ngrok client talking to an ngrok server all locally under your control. Happy hacking!
Network protocol and tunneling
At a high level, ngrok's tunneling works as follows:
Connection Setup and Authentication
- The client initiates a long-lived TCP connection to the server over which they will pass JSON instruction messages. This connection is called the Control Connection.
- After the connection is established, the client sends an Auth message with authentication and version information.
- The server validates the client's Auth message and sends an AuthResp message indicating either success or failure.
- The client may then ask the server to create tunnels for it by sending ReqTunnel messages.
- When the server receives a ReqTunnel message, it will send 1 or more NewTunnel messages that indicate successful tunnel creation or indicate failure.
- When the server receives a new public connection, it locates the appropriate tunnel by examining the HTTP host header (or the port number for TCP tunnels). This connection from the public internet is called a Public Connection.
- The server sends a ReqProxy message to the client over the control connection.
- The client initiates a new TCP connection to the server called a Proxy Connection.
- The client sends a RegProxy message over the proxy connection so the server can associate it to a control connection (and thus the tunnels it's responsible for).
- The server sends a StartProxy message over the proxy connection with metadata information about the connection (the client IP and name of the tunnel).
- The server begins copying the traffic byte-for-byte from the public connection to the proxy connection and vice-versa.
- The client opens a connection to the local address configured for that tunnel. This is called the Private Connection.
- The client begins copying the traffic byte-for-byte from the proxied connection to the private connection and vice-versa.
Detecting dead tunnels
- In order to determine whether a tunnel is still alive, the client periodically sends Ping messages over the control connection to the server, which replies with Pong messages.
- When a tunnel is detected to be dead, the server will clean up all of that tunnel's state and the client will attempt to reconnect and establish a new tunnel.
Messages are sent over the wire as netstrings of the form:
<message length><message payload>
The message length is sent as a 64-bit little endian integer.
The definitions and shared protocol routines lives under src/ngrok/msg
All of the different message types (Auth, AuthResp, ReqTunnel, RegProxy, StartProxy, etc) are defined here and their fields documented. This is a good place to go to understand exactly what messages are sent between the client and server.
ngrokd - the server
Code for the server lives under src/ngrok/server
The ngrokd entry point is in src/ngrok/server/main.go. There is a stub at src/ngrok/main/ngrokd/ngrokd.go for the purposes of creating a properly named binary and being in its own "main" package to comply with go's build system.
ngrok - the client
Code for the client lives under src/ngrok/client
The ngrok entry point is in src/ngrok/client/main.go. There is a stub at src/ngrok/main/ngrok/ngrok.go for the purposes of creating a properly named binary and being in its own "main" package to comply with go's build system.
More documentation can be found in the comments of the code itself.