Skip to content

Commit

Permalink
docs(connectors): add section about adding built-in connectors (#1021)
Browse files Browse the repository at this point in the history
* docs(connectors): add section about adding built-in connectors

* docs(connectors): update wording for conn. to be built in

Co-authored-by: Lovro Mažgon <lovro.mazgon@gmail.com>

* docs(connectors): update instructions for building in connectors

Co-authored-by: Lovro Mažgon <lovro.mazgon@gmail.com>

* docs(connectors): generalize built-in connector source

Co-authored-by: Lovro Mažgon <lovro.mazgon@gmail.com>

* docs(connector): use correct wording for dispenser factory

Co-authored-by: Lovro Mažgon <lovro.mazgon@gmail.com>

* docs(connectors): update code example to use diff highlighting

---------

Co-authored-by: Lovro Mažgon <lovro.mazgon@gmail.com>
  • Loading branch information
jayjayjpg and lovromazgon committed May 4, 2023
1 parent 3641077 commit 07ce9bd
Showing 1 changed file with 71 additions and 9 deletions.
80 changes: 71 additions & 9 deletions docs/connectors.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,18 @@
# Conduit Connectors

Connectors are an integral part of Conduit. Conduit ships with a couple of connectors that are built into the service
to help developers bootstrap pipelines much more quickly. The built-in connectors include Postgres, File, Random Data
Generator, Kafka and Amazon S3 connectors.
Connectors are an integral part of Conduit. Conduit ships with a couple of
[connectors that are directly built into](#built-in-connectors) the service,
but it can also be expanded with additional
[standalone connectors](#standalone-connectors).

The difference between Conduit connectors and those you might find from other services is that Conduit connectors are
Change Data Capture-first (CDC). CDC allows your pipeline to only get the changes that have happened over time instead
of pulling down an entire upstream data store and then tracking diffs between some period of time. This is critical for
building real-time event-driven pipelines and applications. But, we'll note where connectors don't have CDC capabilities.
One of the main differences between Conduit connectors and those that you might find from other services is
that all Conduit connectors are Change Data Capture-first (CDC).

CDC allows your pipeline to only get the changes that have happened over time
instead of pulling down an entire upstream data store and then tracking diffs
between some period of time.
This is critical for building real-time, event-driven pipelines and applications.
But, we'll note where connectors do or do not have CDC capabilities.

## Roadmap & Feedback

Expand All @@ -18,12 +23,69 @@ Give the issue a `+1` if you really need that connector. The upvote will help th
particular connector. If you find that an issue hasn't been created for your data store, please create a new issue in
the Conduit repo.

## Connectors
## More about Connectors

### Built-in connectors

To help developers bootstrap pipelines much more quickly,
Conduit ships with several built-in connectors by default.
This includes the Postgres, File, Random Data Generator, Kafka and Amazon S3 connectors.

If you are creating and distributing customized, pre-built Conduit binaries yourself,
you may want to modify the list of built-in connectors, too.
By modifying the list of built-in connectors,
your compiled binary will include a set of pre-installed connectors specific to your end users' needs,
and it can be run by others, as is, without them having to follow any additional installation instructions.

If you want to create your own Conduit binary with a different set of built-in connectors,
you can build-in an [existing connector](#the-list).
Alternatively, you can also create your own
[using the Conduit connector template or the Conduit Connector SDK](https://conduit.io/docs/connectors/building/).

Once you have chosen a connector to be built-in, you can:

- Download the new package and its dependencies: `go get "github.com/foo/conduit-connector-new"`
- Import the Go module defining the connector
into the [builtin registry](https://github.com/ConduitIO/conduit/blob/main/pkg/plugin/builtin/registry.go)
and add a new key to `DefaultDispenserFactories`:

```diff
package builtin

import (
// ...
file "github.com/conduitio/conduit-connector-file"
// ...
+ myNewConnector "github.com/foo/conduit-connector-new"
)

var (
// DefaultDispenserFactories contains default dispenser factories for
// built-in plugins. The key of the map is the import path of the module
// containing the connector implementation.
DefaultDispenserFactories = map[string]DispenserFactory{
"github.com/conduitio/conduit-connector-file": sdkDispenserFactory(file.Connector),
// ...
+ "github.com/foo/conduit-connector-new": sdkDispenserFactory(myNewConnector.Connector),
}
)
```

- Run `make`
- You now have a binary with your own, custom set of built-in connectors! 🎉

### Standalone connectors

In addition to built-in connectors, Conduit can be used together with standalone connectors
to enable even more data streaming use cases.
The Conduit team and other community developers create and maintain standalone connectors.

Learn more about how you can install [standalone connectors to Conduit here](https://conduit.io/docs/connectors/installing).

### Support Types

- `Conduit` - These are connectors that are built by the Conduit. Any issues or problems filed on those repos will be
respond to by the Conduit team.
responded to by the Conduit team.
- `Community` - A community connector is one where a developer created a connector and they're the ones supporting it
not the Conduit team.
- `Legacy` - Some connectors are built using non-preferred methods. For example, Kafka Connect connectors can be used
Expand Down

0 comments on commit 07ce9bd

Please sign in to comment.