envoy VCL interface registered faild with Vpp

[YanLongDai]

Title: envoy 1.21.6 VCl can‘t work with vpp 21.10，I suspect there is a problem with the compilation process envoy，Please take a look ， I need help， thank you

start envoy log：

vpp log：

vcl.conf :

vcl {
  rx-fifo-size 400000
  tx-fifo-size 400000
  app-scope-global
  api-socket-name /tmp/vpp-api.sock
  use-mq-eventfd
}

**envoy proxy.yaml:**

 admin:
  access_log_path: /tmp/envoy.log
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 8081
static_resources:
  listeners:
  # define a reverse proxy on :10001 that always uses :80 as an origin.
  - address:
      socket_address:
        protocol: TCP
        address: 0.0.0.0
        port_value: 10001
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: ingress_http
          route_config:
            name: local_route
            virtual_hosts:
            - name: service
              domains: ["*"]
              routes:
              - match:
                  prefix: "/"
                route:
                  cluster: proxy_service
          http_filters:
          - name: envoy.filters.http.router
  clusters:
  - name: proxy_service
    connect_timeout: 0.25s
    type: LOGICAL_DNS
    dns_lookup_family: V4_ONLY
    lb_policy: ROUND_ROBIN
    load_assignment:
      cluster_name: proxy_service
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 10.10.1.1
                port_value: 80
bootstrap_extensions:
  - name: envoy.extensions.vcl.vcl_socket_interface
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.vcl.v3alpha.VclSocketInterface
default_socket_interface: "envoy.extensions.vcl.vcl_socket_interface"

Envoy Compile and run steps：

1、bazel build -c dbg --verbose_failures --verbose_explanations --config=clang //contrib/exe:envoy-static --sandbox_debug
2、start_envoy.sh:

#!/usr/bin/env bash
BIN_TYPE=k8-dbg
ENVOY_CFG=configs/proxy.yaml
VCL_CFG=configs/vcl_2.conf

sudo taskset -c 1 sh -c "VCL_CONFIG=$VCL_CFG /home/daiyl/01_work/envoy_vpp/envoy/bazel-bin/contrib/exe/envoy-static --concurrency 1 -c $ENVOY_CFG  --log-level debug

3、VPP Compile and run separately，According to the official website

unix {
  nodaemon
  interactive
  exec configs/vpp.cli
  log /tmp/vpp.log
  full-coredump
  #cli-listen /run/vpp/cli.sock
  cli-listen 127.0.0.1:5002 
  gid vpp
}

heapsize 4g
api-trace {
  on
}

api-segment {
  global-size 2G
  api-size 1G
  gid vpp
}

socksvr {
  socket-name /tmp/vpp-api.sock
}

cpu {
  skip-cores 0
  main-core 1

  ## Set logical CPU core(s) where worker threads are running. For performance testing make
  ## sure the cores are on the same numa as the NIC(s). Use lscpu to determine the numa of
  ## a cpu and "sh hardware" in vpp cli to determine the numa of a NIC
  corelist-workers 2
}

buffers {
  buffers-per-numa 16384
}

dpdk {
  # Note:
  # - Assuming only one NIC
  # - The PCI address is an example, the actual one should be found using dpdk_devbind
  #    https://github.com/DPDK/dpdk/blob/main/usertools/dpdk-devbind.py
  # - Number of rx queues (num-rx-queus should be number of workers + 1)
#  dev 0000:00:18.0 {
#    num-tx-desc 256
#    num-rx-desc 256
#    num-rx-queues 2
#  }
 # dev 0000:00:13.0 {
 #   num-tx-desc 256
 #   num-rx-desc 256
 #   num-rx-queues 2
#  }

}

session {
  evt_qs_memfd_seg
  event-queue-length 100000
}

[YanLongDai]

@ravenblackx

[kyessenov]

CC @florincoras @KfreeZ

[florincoras]

@daiyl123 as per recommended config here could you try using the socket api, as opposed to the binary api, in vcl.conf?

In particular, add use-app-socket-api to session stanza in vpp's startup.conf, remove api-socket-name and add app-socket-api /var/run/vpp/app_ns_sockets/default to vcl.conf.

As a side note, the vcl socket interface envoy builds statically links in vcl from vpp 22.02. If you want to change that, see here

[YanLongDai]

@florincoras @KfreeZ I tried according to your suggestion, but another problem occurred: vpp directly core dump, I change vcl.conf and vpp’s startup.conf;

In addition, My version is envoy 1.21.6，The specified vpp version is 21.10 in repository_locations.bzl

envoy core dump log：
envoy_core_dump.log

vcl.conf：

vpp‘s startup.conf:

vpp’s api sock in environment：

[florincoras]

Note that you haven't switched vcl.conf to app-socket-api /var/run/vpp/app_ns_sockets/default as you're still using the binary api api.sock.

As for vpp version, I'm not sure how well that would work. The current vcl socket interface, which relies on vpp 22.02, is known to work with with vpp latest (23.02), so I'd recommend trying that if everything else fails.

[YanLongDai]

@florincoras Sorry， I don't quite understand， I modified the configuration vcl.conf by referring to our official website

I'm not sure whether I should use Configuration ”app-socket-api /var/run/vpp/app_ns_sockets/default“ or Configuration ”app-socket-api /var/run/vpp/api.sock“

I'm sorry, I don't understand this. I hope you can help me again

If configuration ”app-socket-api /var/run/vpp/app_ns_sockets/default“ is used, an error will also be reported：

vcl {
  # Max rx/tx session buffers sizes in bytes. Increase for high throughput traffic.
  rx-fifo-size 400000
  tx-fifo-size 400000

  # Size of shared memory segments between VPP and VCL in bytes
  segment-size 1000000000

  # App has access to global routing table
  app-scope-global

  # Allow inter-app shared-memory cut-through sessions
  app-scope-local

  # Pointer to session layer's socket api socket
   app-socket-api /var/run/vpp/app_ns_sockets/default

  # Message queues use eventfds for notifications
  use-mq-eventfd

  # VCL worker incoming message queue size
  event-queue-size 40000
}

[YanLongDai]

How to configure vcl.conf to not apply to the binary api api.sock

You said that I was wrong to use configuration ”app-socket-api /var/run/vpp/api.sock“

[YanLongDai]

I use vpp 23.02，The same error is reported，I suspect it is the problem of compiling vpp or vcl.conf

vcl {
  # Max rx/tx session buffers sizes in bytes. Increase for high throughput traffic.
  rx-fifo-size 400000
  tx-fifo-size 400000

  # Size of shared memory segments between VPP and VCL in bytes
  segment-size 1000000000

  # App has access to global routing table
  app-scope-global

  # Allow inter-app shared-memory cut-through sessions
  app-scope-local

  # Pointer to session layer's socket api socket
  #app-socket-api /var/run/vpp/api.sock
  app-socket-api /var/run/vpp/app_ns_sockets/default

  # Message queues use eventfds for notifications
  use-mq-eventfd

  # VCL worker incoming message queue size
  event-queue-size 40000
}

[YanLongDai]

@florincoras
I'm not sure about the purpose of these libraries compiled by envoy. I compiled vpp directly from the vpp source code downloaded from the official website. Do I still need to copy envoy's libraries into vpp

[florincoras]

/var/run/vpp/app_ns_sockets/default is created by vpp only when use-app-socket-api is configured in session stanza. This is the socket vcl uses to find and attach to vpp's host stack. If it doesn't exist, especially when running vpp 23.02, probably vpp did not start correctly.

[florincoras]

Those libraries are built and statically linked into envoy as opposed to being dynamically linked. That is, unfortunately, you can't use .so files published by vpp project or that you manually built.

[YanLongDai]

@florincoras
The problem of envoy vcl startup failure that occurred before is because the use-app-socket-api session of vpp is disabled by default. The vpp startup.conf provided by our envoy official website user manual also lacks an enable configuration item. Can I mention a code merge request?

[florincoras]

@YanLongDai go for it. Just for completeness, session layer in vpp can also be enable via cli, i.e., session enable.

[YanLongDai]

@YanLongDai go for it. Just for completeness, session layer in vpp can also be enable via cli, i.e., session enable.

@florincoras What you said makes sense. PR has been submitted, please help to review #29022