Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OpenTelemetry Protocol with Apache Arrow Exporter initial skeleton #30619

Merged
merged 23 commits into from
Jan 23, 2024

Conversation

jmacd
Copy link
Contributor

@jmacd jmacd commented Jan 16, 2024

Description: First PR to introduce the OpenTelemetry Protocol with Apache Arrow exporter. From the upstream repository: https://github.com/open-telemetry/otel-arrow/blob/main/collector/exporter/otelarrowexporter

Link to tracking Issue: #26491

Testing:
This is a skeleton PR, therefore only the skeleton contains tests. Compared with the upstream repository, the factory_test.go and config_test.go files have been kept, the implementation tests in otelarrow_test.go were excluded in this PR.

Documentation: New README, copied from the upstream repository.

@jmacd jmacd marked this pull request as ready for review January 20, 2024 00:50
@jmacd jmacd requested a review from a team as a code owner January 20, 2024 00:50
@jmacd jmacd requested a review from songy23 January 20, 2024 00:50
…tor-contrib into jmacd/otelarrow_skel_exporter
Copy link
Contributor

@codeboten codeboten left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @jmacd! This looks good, just a few comments


require (
github.com/open-telemetry/otel-arrow v0.16.0
github.com/open-telemetry/otel-arrow/collector v0.16.0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these dependencies will eventually go away once the implementation is complete in this repo correct?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The otel-arrow module contains a large piece of code for translating between Arrow and pdata OTLP. This library probably has uses outside of the Collector and I think it should remain separate, at least until the Arrow proto definitions are accepted formally as OTLP protocol. There are OTel-Go SDK reasons we might use that library outside of collector components, probably.

The otel-arrow/collector module contains a number of components that were useful for us during development of the primary exporter/receiver, and we will want to keep them. They could each move into the -contrib repository, but that will take some additional effort, so I expect the repository itself to remain.

In addition to those statements, there are three packages for which an answer is needed, these being shared by the exporter and receiver and things I would recommend for the core collector itself.

The otel-arrow/collector/sharedcomponent package is a copy of the one from the core repo's internal/ package. I'm not sure why the core repo won't export that functionality, so I replicated it.

The otel-arrow/collector/netstats package is a set of very useful metrics that the Arrow components use, but which are applicable to all components and would also useful for the automatic benchmark harness.

The otel-arrow/collector/compression package is an implementation that supports multiple configurable Zstd compression levels on a per-exporter component basis. See mostynb/go-grpc-compression#24.

I suspect you will agree that all three of these packages belong in either -collector or -collector-contrib.

exporter/otelarrowexporter/metadata.yaml Show resolved Hide resolved
exporter/otelarrowexporter/internal/arrow/exporter.go Outdated Show resolved Hide resolved
exporter/otelarrowexporter/internal/arrow/exporter.go Outdated Show resolved Hide resolved
exporter/otelarrowexporter/config.go Outdated Show resolved Hide resolved
exporter/otelarrowexporter/README.md Show resolved Hide resolved
exporter/otelarrowexporter/README.md Show resolved Hide resolved
Copy link
Contributor

@codeboten codeboten left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for responding to my comments @jmacd, looks good

@codeboten codeboten merged commit f84f48e into open-telemetry:main Jan 23, 2024
85 checks passed
@github-actions github-actions bot added this to the next release milestone Jan 23, 2024
cparkins pushed a commit to AmadeusITGroup/opentelemetry-collector-contrib that referenced this pull request Feb 1, 2024
…pen-telemetry#30619)

First PR to introduce the OpenTelemetry Protocol with
Apache Arrow exporter. From the upstream repository:
https://github.com/open-telemetry/otel-arrow/blob/main/collector/exporter/otelarrowexporter

**Link to tracking Issue:**  open-telemetry#26491 

This is a skeleton PR, therefore only the skeleton contains tests.
Compared with the upstream repository, the `factory_test.go` and
`config_test.go` files have been kept, the implementation tests in
`otelarrow_test.go` were excluded in this PR.

New README, [copied from the upstream
repository](https://github.com/open-telemetry/otel-arrow/blob/main/collector/exporter/otelarrowexporter/README.md).
codeboten pushed a commit that referenced this pull request Feb 7, 2024
…30766)

First PR to introduce the OpenTelemetry Protocol with Apache Arrow
receiver. From the upstream repository:
https://github.com/open-telemetry/otel-arrow/blob/main/collector/receiver/otelarrowreceiver.

Similar to
#30619
for the corresponding receiver.

**Link to tracking Issue:** #26491

**Testing:** 
This is a skeleton PR, therefore only the skeleton contains tests.
Compared with the upstream repository, the factory_test.go and
config_test.go files have been kept, the implementation tests in
otelarrow_test.go were excluded in this PR.

**Documentation:** New README, [copied from the upstream
repository](https://github.com/open-telemetry/otel-arrow/blob/main/collector/receiver/otelarrowreceiver/README.md).

---------

Signed-off-by: Joshua MacDonald <josh.macdonald@gmail.com>
anthoai97 pushed a commit to anthoai97/opentelemetry-collector-contrib that referenced this pull request Feb 12, 2024
…pen-telemetry#30619)

First PR to introduce the OpenTelemetry Protocol with
Apache Arrow exporter. From the upstream repository:
https://github.com/open-telemetry/otel-arrow/blob/main/collector/exporter/otelarrowexporter

**Link to tracking Issue:**  open-telemetry#26491 

This is a skeleton PR, therefore only the skeleton contains tests.
Compared with the upstream repository, the `factory_test.go` and
`config_test.go` files have been kept, the implementation tests in
`otelarrow_test.go` were excluded in this PR.

New README, [copied from the upstream
repository](https://github.com/open-telemetry/otel-arrow/blob/main/collector/exporter/otelarrowexporter/README.md).
anthoai97 pushed a commit to anthoai97/opentelemetry-collector-contrib that referenced this pull request Feb 12, 2024
…pen-telemetry#30766)

First PR to introduce the OpenTelemetry Protocol with Apache Arrow
receiver. From the upstream repository:
https://github.com/open-telemetry/otel-arrow/blob/main/collector/receiver/otelarrowreceiver.

Similar to
open-telemetry#30619
for the corresponding receiver.

**Link to tracking Issue:** open-telemetry#26491

**Testing:** 
This is a skeleton PR, therefore only the skeleton contains tests.
Compared with the upstream repository, the factory_test.go and
config_test.go files have been kept, the implementation tests in
otelarrow_test.go were excluded in this PR.

**Documentation:** New README, [copied from the upstream
repository](https://github.com/open-telemetry/otel-arrow/blob/main/collector/receiver/otelarrowreceiver/README.md).

---------

Signed-off-by: Joshua MacDonald <josh.macdonald@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants