EBS volume metrics collector daemon

[xxx0624]

Summary

1. Add source code of the GRPC server hosted in the new daemon, EBS volume metrics collector, to the common library.
2. Update the common lib vendor.
3. Update agent vendor.

Implementation details

This daemon will host an unix socket based GRPC server which will implement one interface https://github.com/container-storage-interface/spec/blob/master/spec.md#nodegetvolumestats and can provide the capability for ECS Agent to get the volume stats for one given mounted EBS volume.

In this PR, it includes the GRPC server related logics only. And the implementation is same as the volume stats implementation of the existing AWS ebs csi driver implementation (https://github.com/kubernetes-sigs/aws-ebs-csi-driver/) except the following items -

• the current implementation supports file based logging however the AWS ebs csi driver doesn't.
  • The klog is the log lib used by both current implementation and the AWS ebs csi driver. But AWS ebs csi driver forces all logs to go to the stdout/stderr.
• the supported flags are different.
  • Flags supported by current implementation: flags from klog + one custom flag
  • Flags supported by the AWS ebs csi driver:

Usage of aws-ebs-csi-driver:
      --aws-sdk-debug-log                    To enable the aws sdk debug log level (default to false).
      --endpoint string                      Endpoint for the CSI driver server (default "unix://tmp/csi.sock")
      --extra-tags mapStringString           Extra tags to attach to each dynamically provisioned resource. It is a comma separated list of key value pairs like '<key1>=<value1>,<key2>=<value2>'
      --extra-volume-tags mapStringString    DEPRECATED: Please use --extra-tags instead. Extra volume tags to attach to each dynamically provisioned volume. It is a comma separated list of key value pairs like '<key1>=<value1>,<key2>=<value2>'
      --http-endpoint :8080                  The TCP network address where the HTTP server for metrics will listen (example: :8080). The default is empty string, which means the server is disabled.
      --k8s-tag-cluster-id string            ID of the Kubernetes cluster used for tagging provisioned EBS volumes (optional).
      --log-flush-frequency duration         Maximum number of seconds between log flushes (default 5s)
      --log-json-info-buffer-size quantity   [Alpha] In JSON format with split output streams, the info messages can be buffered for a while to increase performance. The default value of zero bytes disables buffering. The size can be specified as number of bytes (512), multiples of 1000 (1K), multiples of 1024 (2Ki), or powers of those (3M, 4G, 5Mi, 6Gi). Enable the LoggingAlphaOptions feature gate to use this.
      --log-json-split-stream                [Alpha] In JSON format, write error messages to stderr and info messages to stdout. The default is to write a single stream to stdout. Enable the LoggingAlphaOptions feature gate to use this.
      --logging-format string                Sets the log format. Permitted formats: "json" (gated by LoggingBetaOptions), "text". (default "text")
      --logtostderr                          log to standard error instead of files. DEPRECATED: will be removed in a future release.
      --user-agent-extra string              Extra string appended to user agent.
  -v, --v Level                              number for the log level verbosity
      --version                              Print the version and exit.
      --vmodule pattern=N,...                comma-separated list of pattern=N settings for file-filtered logging (only works for text log format)
      --volume-attach-limit int              Value for the maximum number of volumes attachable per node. If specified, the limit applies to all nodes. If not specified, the value is approximated from the instance type. (default -1)
      --warn-on-invalid-tag                  To warn on invalid tags, instead of returning an error

how go.mod and go.sum are updated

• Run go mod tidy and go mod vendor in both agent dir and ecs-agent dir already.
• Verified that all new direct dependencies added to go.mod are same as the go.mod of agent.

how to start up the process (won't work until one following PR is out)

# This command below will force all logs (whose log level is not higher than 5) to the specified location
# Please refer this log conventions https://github.com/kubernetes/community/blob/master/contributors/devel/sig-instrumentation/logging.md#what-method-to-use

"/bin/csi-driver" "--endpoint=unix://tmp/csi.sock" "--log_file=/tmp/server.log" "-v=5" "--logtostderr=false" "--stderrthreshold=3"

Testing

New tests cover the changes: yes

Also

• manually validated all tests which are included in the new directory are pass via make test under the new directory.

Description for the changelog

Add the EBS volume metrics collector.

Licensing

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[mye956]

the supported flags are different

nit: Could we include what flags were added/removed/modified somewhere within the PR description?

[mye956]

the current implementation support file based logging via klog however the AWS ebs csi driver doesn't. The klog is the log lib used by the AWS ebs csi driver.

nit(non-blocking): this is a bit confusing to read at first, perhaps this could be reworded better to clarify whether or not klog is used by the AWS ebs csi driver?

[xxx0624]

the supported flags are different

nit: Could we include what flags were added/removed/modified somewhere within the PR description?

Done.

the current implementation support file based logging via klog however the AWS ebs csi driver doesn't. The klog is the log lib used by the AWS ebs csi driver.

nit(non-blocking): this is a bit confusing to read at first, perhaps this could be reworded better to clarify whether or not klog is used by the AWS ebs csi driver?

Sure, updated.