[DISCUSS] proposal: move connectors into a dedicated iggy-connectors repository

[hubcio]

I want to move the whole connectors subsystem (core/connectors/: runtime, sdk,
all sinks and sources + tests) out into its own apache/iggy-connectors repo. nothing
decided yet, just want to hear what people think.

the reason is CI time. our test scoping walks the workspace dependency graph, and any
change to iggy-server ends up pulling the whole rust workspace in, so every
connector gets built and tested too. the plugins are cdylib crates that ci
always builds, so even a one-line server change rebuilds every connector .so
and spins up the docker/testcontainers backends. we have ~16 connector crates
now (10 sinks, 4 sources), some pulling huge trees like arrow/parquet, deltalake
and cloud sdks, mongo, elasticsearch. so people working on the server keep paying
for connectors they never touched, and it gets worse every time someone adds
another heavy one.

splitting is safe because the runtime only talks to the server as a normal
client, over iggy and iggy_binary_protocol, never server internals. testing
connectors against a released server is the same thing real users do. the only
internal contract is the ffi plugin abi in iggy_connector_sdk, and that stays
in the new repo next to the plugins that depend on it.

what we'd get: server CI stops building and container-testing connectors, the
heavy deps leave the main build, and connectors can release on their own clock.
what we'd have to deal with: server changes that break a connector get caught
later (on a release or nightly, not in the pr); iggy_connector_sdk becomes a
real published crate with an abi/semver policy to keep; the crates connectors
depend on (iggy, iggy_common, iggy_binary_protocol, configs) need to be
published and pinned; and since we're incubating, a new repo means going through
asf infra.

rough order if we do it: land the open connector prs, file an apache infra
ticket to create the apache/iggy-connectors repo, split with history into it,
publish and pin the dep crates, set up ci that runs against the latest released
server, then drop core/connectors/ from the main workspace.

a few of us already like the idea, but i'd rather argue it out here first. if
you maintain or run a connector, or you think this is a bad move, say so.

[kparisa]

+1 (Apache Iggy PPMC)

Also, I think we should also create an iggy-example-apps (or just iggy-examples, but there is /examples folder at the moment) repository where the community can contribute real-world use cases and example implementations.

What are everyone’s thoughts on the current /examples folder? currently, they are focused on SDK usage across different languages. Should we keep those SDK-level examples in the main repo and move broader iggy-example repository?

[numinnex]

+1 binding (Apache Iggy PPMC)

I think moving connectors to separate repo would happen sooner or later (number of supported connectors started to grow really fast). Regarding examples, I think those examples that we currently have should stay local to where SDK lives (they are very simple demonstration of basic functionalities), but I don't mind creating a dedicated repo with more detailed applications (we already have couple of those community made, those could be contributed to the new repo).

[ryerraguntla]

+1 for separate connectors.
Couple of questions.

1. Apart from naming convention, is there a way to group these repositories as related and dependent?
2. With separate repositories, How are the milestone releases are managed and published for the complete Product?

[ryankert01]

+1 (non binding)

[mmodzelewski]

+1 I think we would only benefit from separating the connectors. It's a clean cut, so it should be fairly straightforward to move.