Skip to content

Update Beam documentation to include Schema type mapping between various SDK languages #21160

@damccorm

Description

@damccorm

Currently documentation regarding how language specific types map to Beam schema types and vice versa are scattered in different locations.

For example,

  • Following describes how Python types map to Beam schema types.

https://beam.apache.org/releases/pydoc/2.32.0/apache_beam.typehints.schemas.html

  •  Beam Programming Guides gives some details regarding types and few examples but this seems to be incomplete.

https://beam.apache.org/documentation/programming-guide/#schemas

  •  Following has a table that describes how types from various languages relate to each other.

https://docs.google.com/document/d/1uu9pJktzT_O3DxGd1-Q2op4nRk4HekIZbzi-0oTAips/edit#bookmark=id.tlciv62z2d9u

  • There's limited documentation on using Python NamedTupes to properly create Beam Rows.

 

Ideally, we should extend the Beam Programming Guide (or create a new guide) to describe currently supported Schema types and how types from various SDK languages map to such types. We should also have enough examples to illustrate the usage.

 

Imported from Jira BEAM-13170. Original Jira may contain additional context.
Reported by: chamikara.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions