Provisioning mode via HAPROXY

[nfmanasd28]

I need to setup NVFlare such that there is HAPROXY as a reverse proxy between the servers and the clients.
This is the current setup I am using:
The server and admin are deployed on same machine, MACHINE_A, HAPROXY on MACHINE_B and 2 clients on MACHINE_A (For testing purposes).

The project.yml is as follows:

api_version: 3
name: nvflare_mtls_haproxy
description: NVIDIA FLARE sample project yaml file

participants:
  # change example.com to the FQDN of the server
  - name: MACHINE_B
    type: server
    org: nvidia
    fed_learn_port: 8004
    admin_port: 8005
  - name: site-1
    type: client
    org: nvidia
  - name: site-2
    type: client
    org: nvidia
  - name: admin@nvidia.com
    type: admin
    org: nvidia
    role: project_admin

# The same methods in all builders are called in their order defined in builders section
builders:
  - path: nvflare.lighter.impl.workspace.WorkspaceBuilder
    args:
      template_file: master_template.yml
  - path: nvflare.lighter.impl.template.TemplateBuilder
  - path: nvflare.lighter.impl.static_file.StaticFileBuilder
  - args:
      # config_folder can be set to inform NVIDIA FLARE where to get configuration
      config_folder: config

      # app_validator is used to verify if uploaded app has proper structures
      # if not set, no app_validator is included in fed_server.json
      # app_validator: PATH_TO_YOUR_OWN_APP_VALIDATOR

      # when docker_image is set to a docker image name, docker.sh will be generated on server/client/admin
      # docker_image:

      # download_job_url is set to http://download.server.com/ as default in fed_server.json.  You can override this
      # to different url.
      # download_job_url: http://download.server.com/

      overseer_agent:
        path: nvflare.ha.dummy_overseer_agent.DummyOverseerAgent
        # if overseer_exists is true, args here are ignored.  Provisioning
        #   tool will fill role, name and other local parameters automatically.
        # if overseer_exists is false, args in this section will be used and the sp_end_point
        # must match the server defined above in the format of SERVER_NAME:FL_PORT:ADMIN_PORT
        # 
        overseer_exists: false
        args:
          sp_end_point: MACHINE_B:8004:8005

  - path: nvflare.lighter.impl.cert.CertBuilder
  - path: nvflare.lighter.impl.signature.SignatureBuilder

My HAPROXY config looks like this:

frontend nvflare_mtls
   mode tcp
   bind :8004 ssl crt CERT_SERVER alpn h2,http/1.1 verify required ca-file CERT_ROOT_CA
   default_backend nvflare_mtls_server

backend nvflare_mtls_server
   mode tcp
   server s1 MACHINE_A:8004

CERT_SERVER: pem certificate which contains the nvflare server certificate and key
CERT_ROOT_CA: ROOT_CA nvflare certificate

After starting the server, when I am trying to start the client,

I get the following logs on HAPROXY:
nvflare_mtls~ nvflare_mtls_server/s2 11/1/12 0 -- 1/1/0/0/0 0/0

and the following in the NVFlare server:
Handshake failed with fatal error SSL_ERROR_SSL: error:100000f7:SSL routines:OPENSSL_in ternal:WRONG_VERSION_NUMBER.

Can you help me tackle this?

[chesterxgchen]

@kkersten @IsaacYangSLA do you know how to set the HAPROXY config as reverse proxy ? I have seen ngnix config.

[IsaacYangSLA]

@nfmanasd28 , I believe in your configuration, the TLS terminates at HAPROXY. However, NVFlare is using mutual TLS, which means the server also need to get client's certificates to verify client's identity. You will need to configure your HAPROXY to support mTLS.

[nfmanasd28]

@IsaacYangSLA, thank you for your quick reply.
I think the verify required ca-file CERT_ROOT_CA entry enables mtls in haproxy.
Ref: mtls in HAPROXY