Make DataType a separate type within Component #10222

ankitpatel96 · 2024-05-25T18:22:26Z

Description

Followup from #10069

The end goal of this PR is to separate DataType from Type. DataType becomes an enum for Traces, Logs, and Metrics.

Context (to quote @mx-psi):

A component.ID currently is a pair of component.Type and a (possibly empty) name for the component (e.g. otlp/name). We use the fact that component.DataType is also a component.Type to also use it for pipeline names (e.g. traces/sampled).

To solve this, I introduce a component.PipelineID as suggested by @TylerHelmuth. This PipelineID is very similar to component.ID - it just only lets you use a DataType as the Type.

The PR is large but hundreds of these changes are just changing test files to use a DataType instead of a Type - I swear its not as bad as it looks. Most of the actual changes are in component, connector and service/internal/graph/

Followup Work:
I wanted to move PipelineID to service/pipelines - but that would create a circular dependency with InstanceID, which uses PipelineID but lives in component. I want to fix that.

I also want to rename DataType to Signal or something like that, and move it out to its own package.

I didn't do either of those things in this PR since it's already huge.

Link to tracking issue

Fixes #9429

Testing

Unit tests

codecov · 2024-05-25T18:28:41Z

Codecov Report

Attention: Patch coverage is 97.89474% with 2 lines in your changes missing coverage. Please review.

Project coverage is 92.53%. Comparing base (effe267) to head (304d0b0).
Report is 54 commits behind head on main.

❗ Current head 304d0b0 differs from pull request most recent head c1252e4

Please upload reports for the commit c1252e4 to get more accurate results.

Files	Patch %	Lines
component/pipeline.go	93.93%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #10222      +/-   ##
==========================================
+ Coverage   92.52%   92.53%   +0.01%     
==========================================
  Files         388      389       +1     
  Lines       18310    18364      +54     
==========================================
+ Hits        16942    16994      +52     
- Misses       1022     1023       +1     
- Partials      346      347       +1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

ankitpatel96 · 2024-05-29T17:10:55Z

This is changing an API - and it seems like I will have to change contrib in lockstep with this change (bunch of tests are failing). Any advice on how to do that would be welcome!

mx-psi · 2024-06-05T11:22:47Z

Requesting review from the reviewers on #10069

component/config.go

component/pipeline.go

mx-psi

Looking at the dependency graph, I think InstanceID should not be part of the service module. Either we make service/pipelines into a separate module or we move this to a separate pipelines module altogether.

mx-psi · 2024-06-11T11:26:24Z

component/component.go

@@ -189,10 +189,3 @@ type CreateDefaultConfigFunc func() Config
 func (f CreateDefaultConfigFunc) CreateDefaultConfig() Config {
 	return f()
 }
-
-// InstanceID uniquely identifies a component instance
-type InstanceID struct {


Can we add a

// Deprecated use [<ref to new type>] type InstanceID = <ref to new type>

here?

Can I? That would make component dependent on pipelines... which is most definitely circular

We can use an internal module and make this and the actual type aliases of a symbol in that internal module until we remove it from component:

// package internal type InstanceID struct { /* actual definition goes here *? }

// package component // Deprecated: [v0.103.0] use pipelines.InstanceID type InstanceID = internal.InstanceID

// package pipelines type InstanceID = internal.InstanceID

I don't think this is going to work -

Pipeline depends on internal, which depends on pipeline since InstanceID depends on pipeline.ID

If I move pipeline.ID into this internal package, we solve that problem but then the component.InstanceID shim doesn't work - that means component imports internal and then internal imports component (InstanceID contains component.ID and component.Kind).

Open to any ideas to fixing this..

Side note - this internal package has to be collector/internal/ since it needs to be accessible by collector/component and collector/pipeline.

A solution here would be to also move component.ID and component.Kind to internal and make shims also for these two on component. It sounds like a pain, so let's first discuss the rest of the PR

Right that would probably work... but I do question whether its worth it for stability concerns. Moving component.ID to internal/ would make component dependent on the root collector module, thus adding that dependency to almost every other module.

I don't think the blast radius of this change is especially large for contrib and I think, in this case, it makes sense to keep this change as is and not try to provide backwards compatible import shims.

ankitpatel96 · 2024-06-12T23:21:24Z

I moved InstanceID and PipelineID into a new pipeline module!

ankitpatel96 · 2024-06-20T18:09:33Z

Bump - would really appreciate a review here before I do another merge to fix all these conflicts

mx-psi

I think this PR makes sense to merge with this public API. There are a bunch of breaking changes which are hard to work around in some instances. IMO we should

document the list of breaking changes on the changelog
evaluate the impact of breaking changes looking at the contrib tests and think about adding two-step deprecation process for the more impactful ones

component/config.go

mx-psi · 2024-06-24T14:09:58Z

component/identifiable.go

-// typeAndNameSeparator is the separator that is used between type and name in type/name composite keys.
-const typeAndNameSeparator = "/"
+// TypeAndNameSeparator is the separator that is used between type and name in type/name composite keys.
+const TypeAndNameSeparator = "/"


Maybe this should be renamed since we also use it for separating DataType and pipeline name. Alternatively, move it to internal and make it private

mx-psi · 2024-06-24T14:11:28Z

connector/logs_router.go

-	Consumer(...component.ID) (consumer.Logs, error)
-	PipelineIDs() []component.ID
+	Consumer(...pipeline.ID) (consumer.Logs, error)
+	PipelineIDs() []pipeline.ID


This (and the ones on metrics and traces router) are breaking changes, it seems hard to work around them so we can just add an API changelog entry

mx-psi · 2024-06-24T14:16:21Z

extension/extensiontest/statuswatcher_extension.go

@@ -21,7 +22,7 @@ func NewStatusWatcherExtensionCreateSettings() extension.Settings {

 // NewStatusWatcherExtensionFactory returns a component.ExtensionFactory to construct a status watcher extension.
 func NewStatusWatcherExtensionFactory(
-	onStatusChanged func(source *component.InstanceID, event *component.StatusEvent),
+	onStatusChanged func(source *pipeline.InstanceID, event *component.StatusEvent),


This is also a breaking change, but if we end up not having a type alias for component.InstanceID, users of this API are going to have a build time error regardless so they will be 'notified' that way. We can list it in the API changelog as a breaking change as well.

mx-psi · 2024-06-24T14:21:50Z

pipeline/pipeline.go

+}
+
+// UnmarshalText implements the encoding.TextUnmarshaler interface.
+func (id *ID) UnmarshalText(text []byte) error {


We could have an internal package that has this implementation and allows it to be reused by both component.ID and pipeline.ID. One way of doing this would be having a function that returns two strings and we can reuse the parsing logic.

I don't know if this is actually desirable - the root module (which internal is a part of) already imports component, so this would create a dependency loop. It shouldn't create an import loop but I'm still not sure if this is desirable.

service/extensions/extensions.go

TylerHelmuth

Finally got a chance to take a look at this, and now I can see that my original suggestion did not factor in InstanceID, which even further complicates this change.

Being able to see this change now in its entirety makes me rethink if this change is valuable. While conceptually I think it is nice, to an end-user this changes nothing. I am concerned about the amount of time we'll spend dealing with this breaking change for what essentially ends up being a rename (a nice rename, but a rename non-the-less.

If, after 1.0, we decide this is valuable, it could be something that is considered for 2.0, but at this point we've been using component.InstanceID, component.Type, and component.DataType successfully without end-users (or component authors) complaining.

At this point I suggest we abandon this change and call #9429 completed as "we decided this wasn't worth it". @ankitpatel96 thank you for your effort on this PR, seeing it all laid out really helped.

mx-psi · 2024-07-08T15:57:58Z

. While conceptually I think it is nice, to an end-user this changes nothing. I am concerned about the amount of time we'll spend dealing with this breaking change for what essentially ends up being a rename (a nice rename, but a rename non-the-less.

This may end up being the case, but thinking about this a bit further, before completely abandoning this I personally want to:

Define for how long we want to support the Go API on different Go modules
Think about how costly different Go API changes will be after 1.0

The first PR related to these two goals is #10539, which will help us then define the support policy for Go API modules.

ankitpatel96 added 11 commits May 24, 2024 17:07

make a new id type with datatype and use it

1329377

PipelineID

4a0e0fa

prune methods

d782e7d

methods

aa519a7

remove unecessary constructor

1904b6d

delete comment

f80c5ec

mass replace id constructors

91bf4f0

switch tests to pipelineid

e5c7f5d

change datagen

e401781

cleanup changes

ab1811d

fix rebase

73bbec2

ankitpatel96 added 8 commits May 28, 2024 11:17

private method

c9180b7

move pipeline id to new file

42dae7d

test coverage

eaa4e20

copyright

cb9e83b

porto

d7422bd

lint

66d375f

chlog

631568c

unit test

77778a9

ankitpatel96 marked this pull request as ready for review May 29, 2024 16:47

ankitpatel96 requested a review from a team as a code owner May 29, 2024 16:47

ankitpatel96 requested a review from atoulme May 29, 2024 16:47

ankitpatel96 changed the title ~~[DRAFT] [NOT READY] Make DataType a separate type within Component~~ Make DataType a separate type within Component May 29, 2024

mx-psi requested review from bogdandrutu, djaglowski, mx-psi and TylerHelmuth June 5, 2024 11:22

mx-psi reviewed Jun 5, 2024

View reviewed changes

component/config.go Outdated Show resolved Hide resolved

component/pipeline.go Outdated Show resolved Hide resolved

djaglowski reviewed Jun 5, 2024

View reviewed changes

component/pipeline.go Outdated Show resolved Hide resolved

ankitpatel96 mentioned this pull request Jun 5, 2024

Make DataType a separate type within Component #10069

Closed

ankitpatel96 force-pushed the datatype-newid branch from f5fe2c1 to 77778a9 Compare June 10, 2024 18:55

ankitpatel96 and others added 5 commits June 10, 2024 14:57

Merge branch 'main' into datatype-newid

4b548a7

fix

149cb87

more merge fixups

5aa24ba

move instanceid

5b96c29

go.mod tidy

304d0b0

mx-psi reviewed Jun 11, 2024

View reviewed changes

ankitpatel96 added 11 commits June 12, 2024 14:51

separator is public

6a94f32

datatype unmarshal

b86c1e4

new module

777bb20

go mod updates

a43fa35

rename pipeline id

929f67a

instance id move

2c1a00b

instanceid go mod

4aec05c

crosslink

dac83e3

trim tags

a302ca1

rename some methods

3676ca0

chloggen

22b2539

mx-psi reviewed Jun 24, 2024

View reviewed changes

ankitpatel96 added 4 commits June 24, 2024 17:59

quote the bad datatype

1475a25

update component tests

534db04

add to changelog

3d82883

license

c1252e4

TylerHelmuth reviewed Jul 3, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make DataType a separate type within Component #10222

Make DataType a separate type within Component #10222

ankitpatel96 commented May 25, 2024 •

edited

Loading

codecov bot commented May 25, 2024 •

edited

Loading

ankitpatel96 commented May 29, 2024

mx-psi commented Jun 5, 2024

mx-psi left a comment

mx-psi Jun 11, 2024

ankitpatel96 Jun 12, 2024

mx-psi Jun 13, 2024

ankitpatel96 Jun 13, 2024

mx-psi Jun 14, 2024

ankitpatel96 Jun 14, 2024

ankitpatel96 commented Jun 12, 2024

ankitpatel96 commented Jun 20, 2024

mx-psi left a comment

mx-psi Jun 24, 2024

mx-psi Jun 24, 2024

mx-psi Jun 24, 2024

mx-psi Jun 24, 2024

ankitpatel96 Jun 24, 2024

TylerHelmuth left a comment

mx-psi commented Jul 8, 2024

Make DataType a separate type within Component #10222

Are you sure you want to change the base?

Make DataType a separate type within Component #10222

Conversation

ankitpatel96 commented May 25, 2024 • edited Loading

Description

Link to tracking issue

Testing

codecov bot commented May 25, 2024 • edited Loading

Codecov Report

ankitpatel96 commented May 29, 2024

mx-psi commented Jun 5, 2024

mx-psi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ankitpatel96 commented Jun 12, 2024

ankitpatel96 commented Jun 20, 2024

mx-psi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TylerHelmuth left a comment

Choose a reason for hiding this comment

mx-psi commented Jul 8, 2024

ankitpatel96 commented May 25, 2024 •

edited

Loading

codecov bot commented May 25, 2024 •

edited

Loading