Running on Heroku

[gmile]

Hi, all!

I'm having an issue with running goreplay on Heroku. Particularly, running goreplay stumbles upon a handful of permission errors like this:

2021-08-10T08:46:51.246738+00:00 app[web.1]: PCAP Activate device error: "Permission Denied", interface: "eth0"
2021-08-10T08:46:51.246739+00:00 app[web.1]: PCAP Activate device error: "Permission Denied", interface: "any"
2021-08-10T08:46:51.246739+00:00 app[web.1]: PCAP Activate device error: "Permission Denied", interface: "lo"
2021-08-10T08:46:51.246740+00:00 app[web.1]: PCAP Activate device error: "Error: NFULNL_CFG_CMD_PF_UNBIND: Operation not permitted", interface: "nflog"
2021-08-10T08:46:51.246740+00:00 app[web.1]: PCAP Activate device error: "Error: NFQNL_CFG_CMD_PF_UNBIND: Operation not permitted", interface: "nfqueue"

Is there a known way to work around these and to intercept HTTP requests?

For context, I am working on an app in Elixir/Erlang, it's currently deployed on Heroku. Ahead of a planned move off Heroku, we are considering replaying existing production traffic to a server in another location. All this to generate data for performance analysis.

There seems to be evidence, that goreplay worked on Heroku in the past in a form of this Heroku buildpack: https://elements.heroku.com/buildpacks/jtrim/heroku-buildpack-traffic-duplicator. Interestingly, there even seem to be "recent deploys" (not sure how recent is that, though).

Buildpack above relies on a flag called --input-http. A quick homework reveals the following:

• this flag was initially requested in Any way this could be used in heroku? #3,
• it was implemented as of HTTP input for listener  #45,
• it was later commented out in f0acd31,
• ultimately it was removed as of Benchmarking, packaging, and fix issues, tests and perfomance #797.

I am now inclined to try an older version of goreplay to test out the flag. @buger would you say it makes sense to do so? If I manage to make it work, maybe it will be good enough for our case.

[gmile]

ℹ️ attempting to an older version of gor on macOS 11.4 (20F71), perhaps expectedly, fails with the following error:

terminated by signal SIGSEGV (Address boundary error)

[buger]

@gmile can you describe how you run it now, embed to app container?

[gmile]

@buger sure

Heroku offers buildpacks and docker containers to deploy the app. We user the former approach, er.g. buildpacks. In particular, our app's deployment is fully within this buildpack: https://github.com/HashNuke/heroku-buildpack-elixir.

Once the application stars, the application code asynchronously downloads goreplay into the file system, and attempts to start it. Respective code in Elixir programming language looks like this:

# this code is executed somewhere early in the web server application boot process

Task.async(fn ->
  Application.ensure_started(:inets)
  Application.ensure_started(:ssl)

  gor_release_url =
    case :os.type() do
      {:unix, :darwin} -> 'https://github.com/buger/goreplay/releases/download/v1.3.0/gor_1.3.0_mac.tar.gz'
      {:unix, :linux} -> 'https://github.com/buger/goreplay/releases/download/v1.3.0/gor_1.3.0_x64.tar.gz'
    end

  {:ok, {_status, _heeaders, binary}} = :httpc.request(:get, {gor_release_url, []}, [autoredirect: true], [])
  {:ok, [{'gor', gor_binary}]} = :erl_tar.extract({:binary, IO.iodata_to_binary(binary)}, [:compressed, :memory])

  executable_path = "#{System.tmp_dir()}/gor"
  :ok = File.write(executable_path, gor_binary)
  {:ok, stat} = File.stat(executable_path)
  :ok = File.write_stat!(executable_path, %{stat | mode: 33261})

  System.cmd(executable_path, ["--input-raw", ":4000", "--output-http=#{System.get_env("MIRROR_TRAFFIC_TO")}"])
end)

We ultimately went with a different approach based on nginx & its mirroring module. I will follow-up with another message detailing our final solution.

[gmile]

Ultimately, our team went with doing mirroring using nginx module called "ngx_http_mirror_module".

Our set up ended up looking like this:

• include nginx buildpack layer to application on Heroku (we had to fork original repo to include our own patch, see below),
• modify our application webserver's configuration to read from a UNIX socket, instead of TCP port,
• enable mirroring in nginx: nginx will now send the request to a) the UNIX socket and b) elsewhere (e.g. a specified URL).

Our patch for nginx config would look like this:

Patch for nginx.conf

diff --git a/config/nginx.conf.erb b/config/nginx.conf.erb
index 85aed6c..5813075 100644
--- a/config/nginx.conf.erb
+++ b/config/nginx.conf.erb
@@ -14,6 +14,7 @@ http {
 	gzip_min_length 512;
 	gzip_proxied any; # Heroku router sends Via header
 
+	mirror http;
 	server_tokens off;
 
 	log_format l2met 'measure#nginx.service=$request_time request_id=$http_x_request_id';
@@ -41,11 +42,30 @@ http {
 			# if ($http_x_forwarded_proto != "https") {
 			#   return 301 https://$host$request_uri;
 			# }
+			<% if ENV['NGINX_MIRROR_TRAFFIC_TO'] %>
+			mirror /mirror;
+			<% end %>
 
 			proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 			proxy_set_header Host $http_host;
 			proxy_redirect off;
 			proxy_pass http://app_server;
+
+			# Preflighted requests
+			if ($request_method = OPTIONS) {
+				add_header "Access-Control-Allow-Origin" "*";
+				add_header "Access-Control-Allow-Methods" "GET, POST, OPTIONS, HEAD, PATCH";
+				add_header "Access-Control-Allow-Headers" "Authorization, Origin, X-Requested-With, Content-Type, Accept";
+				return 200;
+			}
+		}
+
+		<% if ENV['NGINX_MIRROR_TRAFFIC_TO'] %>
+		location = /mirror {
+			internal;
+			resolver 8.8.8.8;
+			proxy_pass https://<%= ENV['NGINX_MIRROR_TRAFFIC_TO'] %>$request_uri;
 		}
+		<% end %>
 	}
 }

[buger]

Makes sense! Thank you for the detailed response!

Indeed looks like for Heroku using current GoReplay approach is dead end.
I thought about making GoReplay a simple HTTP proxy for such cases, but not sure it will be common use-case.