Issues using PUBLIC_PROTOCOL

[klingenm]

Hi!

I'm running bors-ng in a Kubernetes (EKS) cluster behind an application load balancer which terminates the HTTPS for me.

I'm trying to get the health checks to properly request /health, but I'm not able to get it to work.

I found the following PR (#1043) which sounds like what I want, but regardless of what I do I just get

13:37:07.016 pid=<0.2342.0> [info] Plug.SSL is redirecting GET /health to https://bors.mydomain with status 301

for any request that does not have the header with value:

• x-forwarded-for: https
• host: localhost

I tried setting the config var PUBLIC_PROTOCOL to http and also changing the PUBLIC_PORT to 80. The only change in startup logs between these settings is the following line:

1. with PUBLIC_PROTOCOL=http and PUBLIC_PORT=80

13:24:49.139 pid=<0.2067.0> [info] Access BorsNG.Endpoint at http://bors.mydomain

2. with PUBLIC_PROTOCOL=http and PUBLIC_PORT=443 (default)

13:23:34.459 pid=<0.2067.0> [info] Access BorsNG.Endpoint at http://bors.mydomain:443

My full startup log looks like this:

+ db_addr=postgres:5432/postgres
+ db_addr=postgres:5432
+ dockerize -wait tcp://postgres:5432 true
2022/08/26 13:43:24 Waiting for: tcp://postgres:5432
2022/08/26 13:43:24 Connected to tcp://postgres:5432
2022/08/26 13:43:24 Command finished successfully.
+ exec ./bors/bin/bors foreground
Running migrations for bors
13:43:43.831 pid=<0.2033.0> [info] The function passed as a handler with ID {Phoenix.Logger, [:phoenix, :channel_handled_in]} is a local function.
This means that it is either an anonymous function or a capture of a function without a module specified. That may cause a performance penalty when calling that handler. For more details see the note in `telemetry:attach/4` documentation.

https://hexdocs.pm/telemetry/telemetry.html#attach/4
13:43:43.831 pid=<0.2033.0> [info] The function passed as a handler with ID {Phoenix.Logger, [:phoenix, :channel_joined]} is a local function.
This means that it is either an anonymous function or a capture of a function without a module specified. That may cause a performance penalty when calling that handler. For more details see the note in `telemetry:attach/4` documentation.

https://hexdocs.pm/telemetry/telemetry.html#attach/4
13:43:43.831 pid=<0.2033.0> [info] The function passed as a handler with ID {Phoenix.Logger, [:phoenix, :endpoint, :start]} is a local function.
This means that it is either an anonymous function or a capture of a function without a module specified. That may cause a performance penalty when calling that handler. For more details see the note in `telemetry:attach/4` documentation.

https://hexdocs.pm/telemetry/telemetry.html#attach/4
13:43:43.831 pid=<0.2033.0> [info] The function passed as a handler with ID {Phoenix.Logger, [:phoenix, :endpoint, :stop]} is a local function.
This means that it is either an anonymous function or a capture of a function without a module specified. That may cause a performance penalty when calling that handler. For more details see the note in `telemetry:attach/4` documentation.

https://hexdocs.pm/telemetry/telemetry.html#attach/4
13:43:43.831 pid=<0.2033.0> [info] The function passed as a handler with ID {Phoenix.Logger, [:phoenix, :error_rendered]} is a local function.
This means that it is either an anonymous function or a capture of a function without a module specified. That may cause a performance penalty when calling that handler. For more details see the note in `telemetry:attach/4` documentation.

https://hexdocs.pm/telemetry/telemetry.html#attach/4
13:43:43.831 pid=<0.2033.0> [info] The function passed as a handler with ID {Phoenix.Logger, [:phoenix, :router_dispatch, :start]} is a local function.
This means that it is either an anonymous function or a capture of a function without a module specified. That may cause a performance penalty when calling that handler. For more details see the note in `telemetry:attach/4` documentation.

https://hexdocs.pm/telemetry/telemetry.html#attach/4
13:43:43.831 pid=<0.2033.0> [info] The function passed as a handler with ID {Phoenix.Logger, [:phoenix, :socket_connected]} is a local function.
This means that it is either an anonymous function or a capture of a function without a module specified. That may cause a performance penalty when calling that handler. For more details see the note in `telemetry:attach/4` documentation.

https://hexdocs.pm/telemetry/telemetry.html#attach/4
13:43:44.266 pid=<0.2100.0> [warning] Phoenix.Endpoint.CowboyAdapter is deprecated and it will be removed in next versions. Please make sure {:plug_cowboy, "~> 2.1"} or later is listed in your mix.exs
13:43:44.269 pid=<0.2067.0> [info] Access BorsNG.Endpoint at http://bors.mydomain:443
13:43:53.854 pid=<0.2228.0> [info] Plug.SSL is redirecting GET /health to https://bors.mydomain with status 301

Last row shows the log from when the Application load balancer makes the health check request, which is redirected to an https url. This happens with all combinations of PUBLIC_PROTOCOL and PUBLIC_PORT.

From digging a bit, it seems the "scheme" for https://hexdocs.pm/plug/Plug.SSL.html always is https regardless of the PUBLIC_PROTOCOL setting.

I would like to dig into this myself, but I have 0 experience with Elixir and not sure if the startup is worth it. I can work around the issue by accepting the 301 as an ok response, but I think this is something others could benefit from as well.

[notriddle]

I think you want to set FORCE_SSL to false?

https://github.com/bors-ng/bors-ng/blob/master/config/prod.secret.exs is the canonical source for most environment variables.

[klingenm]

I was just writing :), I found https://github.com/bors-ng/bors-ng/blob/master/config/prod.secret.exs and deduced that would be what I wanted, but still same...

Right now my config vars look like this (env vars generated using kustomize)

configMapGenerator:
  - name: bors-ng
    literals:
      - DATABASE_USE_SSL=false
      - FORCE_SSL=false
      - PUBLIC_PROTOCOL=http
      - PUBLIC_PORT=80
      - PUBLIC_HOST=bors.***
      - MIX_ENV=prod
      - GITHUB_CLIENT_ID=****
      - GITHUB_INTEGRATION_ID=****** # NOTE Github App ID
      - DATABASE_URL=postgresql://postgres:postgres@postgres:5432/postgres

I also used a local shell to confirm all env vars are set properly.

[notriddle]

If it still doesn’t work, try building your own Docker image with line 37 (the one mentioning :x_forwarded_proto) removed. Also remove the trailing comma.

[klingenm]

If it still doesn’t work, try building your own Docker image with line 37 (the one mentioning :x_forwarded_proto) removed.

Thanks! I'll do that. But that will be next week. Reaching EOD here.

[jscarrott]

I had the exact same issue and removing the line fixed it for me

[prvnmali2017]

it seems like Bors-ng is trying to redirect the health check requests to HTTPS even though your load balancer is handling the HTTPS termination.

You can try setting the PUBLIC_PROTOCOL environment variable to http to configure Bors-ng to use HTTP for public endpoints. Additionally, you may need to set the PUBLIC_PORT environment variable to 80 to match the port being used by your load balancer.

It may also be helpful to check the headers being sent in the health check request and ensure that they match what Bors-ng is expecting. The logs indicate that Bors-ng is expecting the x-forwarded-for header to be set to https and the host header to be set to localhost. Make sure these headers are set correctly in the health check requests.