Skip to content

[Feature Request]: Add User Worker Threads info to Dataflow backend #26726

@EllaDing

Description

@EllaDing

What would you like to happen?

Reporting some signals from legacy user workers to Dataflow backend for internal autotuning policies to use. The signals planned to be plumbed:

  1. number of active threads
  2. outstanding bytes & its limit
  3. outstanding number of bundles & its limit

Issue Priority

Priority: 3 (nice-to-have improvement)

Issue Components

  • Component: Python SDK
  • Component: Java SDK
  • Component: Go SDK
  • Component: Typescript SDK
  • Component: IO connector
  • Component: Beam examples
  • Component: Beam playground
  • Component: Beam katas
  • Component: Website
  • Component: Spark Runner
  • Component: Flink Runner
  • Component: Samza Runner
  • Component: Twister2 Runner
  • Component: Hazelcast Jet Runner
  • Component: Google Cloud Dataflow Runner

Metadata

Metadata

Assignees

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions