Permalink
Browse files

First pass for router v2 implementation. The init version of router v…

…2 is done by

Jun Xiao, polished by Anfernee with a couple of bug fixes and enhancement.

router v2 decouples control and data path to some extent, it's a logic router
including both nginx and uls (upstream location server) parts.

uls (aka control plane) is based on legacy router:
- all communications with nats are kept the same as before
- client facing interface (actually connected to nginx) is simplified to
  handle the location query/stats update from nginx only.
- the original app facing interface is totally removed.

nginx (aka data plane)
- package lua in nginx so most functionalities can be implemented with it.
- for each request, nginx will generate a subrequest (to fit in nginx
  single thread event model) which carries location query/stats info to uls,
  the response of the subrequest will return backend addr and also an
  opaque tags (only known by uls), and the corresponding main request will
  also be awaked and routed to the backend server, when a response from backend
  is returned back, the counter associated with the particular opaque tags
  will be incremented accoring to the response code. The next incoming
  request will always help to carry not-synced stats to uls along with
  its location query.
- sticky session and trace header are also implemented here,
  also their key related config are moved to nginx part.

testing
- "rake spec" will only test uls, while "rake test" will run both unit test and
  integration test with nginx
- integration test requires nginx with lua module installed running

Change-Id: I9b25b24c449b0421754e541ab5ba60a00c551da9
  • Loading branch information...
1 parent 9fde59f commit 3fec163dcd66b075f9bb8cf9ce58d70777a29ab9 @anfernee anfernee committed Dec 15, 2011
View
@@ -49,7 +49,11 @@ namespace "test" do
sh("cd spec && rake spec")
end
- task "spec:rcov" do |t|
+ task "test" do |t|
+ sh("cd spec && rake test")
+ end
+
+ task "spec:rcov" do |t|
sh("cd spec && rake spec:rcov")
end
end
View
@@ -21,7 +21,6 @@
require 'router/const'
require 'router/router'
-require 'router/app_connection'
require 'router/client_connection'
require 'router/utils'
@@ -183,4 +182,3 @@
end
end
-
@@ -1,200 +0,0 @@
-# Copyright (c) 2009-2011 VMware, Inc.
-module AppConnection
-
- attr_reader :outstanding_requests
-
- def initialize(client, request, droplet)
- Router.log.debug "Creating AppConnection"
- @client, @request, @droplet = client, request, droplet
- @start_time = Time.now
- @connected = false
- @outstanding_requests = 1
- Router.outstanding_request_count += 1
- end
-
- def post_init
- VCAP::Component.varz[:app_connections] = Router.app_connection_count += 1
- Router.log.debug "Completed AppConnection"
- Router.log.debug Router.connection_stats
- Router.log.debug "------------"
- end
-
- def connection_completed
- @connected = true
- #proxy_incoming_to(@client) if @client
- send_data(@request) if @client && @request
- end
-
- # queue data until connection completed.
- def deliver_data(data)
- return send_data(data) if @connected
- @request << data
- end
-
- # We have the HTTP Headers complete from the client
- def on_headers_complete(headers)
- check_sticky_session = STICKY_SESSIONS =~ headers[SET_COOKIE_HEADER]
- sent_session_cookie = false # Only send one in case of multiple hits
-
- header_lines = @headers.split("\r\n")
- header_lines.each do |line|
- @client.send_data(line)
- @client.send_data(CR_LF)
- if (check_sticky_session && !sent_session_cookie && STICKY_SESSIONS =~ line)
- sid = Router.generate_session_cookie(@droplet)
- scl = line.sub(/\S+\s*=\s*\w+/, "#{VCAP_SESSION_ID}=#{sid}")
- sent_session_cookie = true
- @client.send_data(scl)
- @client.send_data(CR_LF)
- end
- end
- # Trace if properly requested
- if @client.trace
- router_trace = "#{VCAP_ROUTER_HEADER}:#{Router.inet}#{CR_LF}"
- be_trace = "#{VCAP_BACKEND_HEADER}:#{@droplet[:host]}:#{@droplet[:port]}#{CR_LF}"
- @client.send_data(router_trace)
- @client.send_data(be_trace)
- end
- # Ending CR_LF
- @client.send_data(CR_LF)
- end
-
- def process_response_body_chunk(data)
- return unless data and data.bytesize > 0
-
- # Let parser process as well to properly determine end of message.
- # TODO: Once EM 1.0, add in optional bytsize proxy if Content-Length is present.
- psize = @parser << data
- if (psize == data.bytesize)
- @client.send_data(data)
- else
- Router.log.info "Pipelined response detected!"
- # We have a pipelined response, we need to hand over the new headers and only send the proper
- # body segments to the backend
- body = data.slice!(0, psize)
- @client.send_data(body)
- receive_data(data)
- end
- end
-
- def record_stats
- return unless @parser
-
- latency = ((Time.now - @start_time) * 1000).to_i
- response_code = @parser.status_code
- response_code_metric = :responses_xxx
- if (200..299).include?(response_code)
- response_code_metric = :responses_2xx
- elsif (300..399).include?(response_code)
- response_code_metric = :responses_3xx
- elsif (400..499).include?(response_code)
- response_code_metric = :responses_4xx
- elsif (500..599).include?(response_code)
- response_code_metric = :responses_5xx
- end
-
- VCAP::Component.varz[response_code_metric] += 1
- VCAP::Component.varz[:latency] << latency
-
- if @droplet[:tags]
- @droplet[:tags].each do |key, value|
- tag_metrics = VCAP::Component.varz[:tags][key][value]
- tag_metrics[response_code_metric] += 1
- tag_metrics[:latency] << latency
- end
- end
- end
-
- def on_message_complete
- record_stats
- @parser = nil
- @outstanding_requests -= 1
- Router.outstanding_request_count -= 1
- :stop
- end
-
- def cant_be_recycled?
- error? || @parser != nil || @connected == false || @outstanding_requests > 0
- end
-
- def recycle
- stop_proxying
- @client = @request = @headers = nil
- end
-
- def receive_data(data)
- # Parser is created after headers have been received and processed.
- # If it exists we are continuing the processing of body fragments.
- # Allow the parser to process to signal proper end of message, e.g. chunked, etc
- return process_response_body_chunk(data) if @parser
-
- # We are awaiting headers here.
- # We buffer them if needed to determine the header/body boundary correctly.
- @buf = @buf ? @buf << data : data
- if hindex = @buf.index(HTTP_HEADERS_END) # all http headers received, figure out where to route to..
- @parser = Http::Parser.new(self)
-
- # split headers and rest of body out here.
- @headers = @buf.slice!(0...hindex+HTTP_HEADERS_END_SIZE)
-
- # Process headers
- @parser << @headers
-
- # Process left over body fragment if any
- process_response_body_chunk(@buf) if @parser
- @buf = @headers = nil
- end
-
- rescue Http::Parser::Error => e
- Router.log.debug "HTTP Parser error on response: #{e}"
- close_connection
- end
-
- def rebind(client, request)
- @start_time = Time.now
- @client = client
- reuse(request)
- end
-
- def reuse(new_request)
- @request = new_request
- @outstanding_requests += 1
- Router.outstanding_request_count += 1
- deliver_data(@request)
- end
-
- def proxy_target_unbound
- Router.log.debug "Proxy connection dropped"
- #close_connection_after_writing
- end
-
- def unbind
- Router.outstanding_request_count -= @outstanding_requests
- unless @connected
- Router.log.info "Could not connect to backend for url:#{@droplet[:url]} @ #{@droplet[:host]}:#{@droplet[:port]}"
- if @client
- @client.send_data(Router.notfound_redirect || ERROR_404_RESPONSE)
- @client.close_connection_after_writing
- end
- # TODO(dlc) fix - We will unregister bad backends here, should retry the request if possible.
- Router.unregister_droplet(@droplet[:url], @droplet[:host], @droplet[:port])
- end
-
- VCAP::Component.varz[:app_connections] = Router.app_connection_count -= 1
- Router.log.debug 'Unbinding AppConnection'
- Router.log.debug Router.connection_stats
- Router.log.debug "------------"
-
- # Remove ourselves from the connection pool
- @droplet[:connections].delete(self)
-
- @client.close_connection_after_writing if @client
- end
-
- def terminate
- stop_proxying
- close_connection
- on_message_complete if @outstanding_requests > 0
- end
-
-end
Oops, something went wrong.

0 comments on commit 3fec163

Please sign in to comment.