Bespoke custom connectors

[oxlade39]

Dozer looks like an amazingly promising tool. I'd like to explore using it in an existing and complex cloud infrastructure.

In our existing cloud infrastructure we have several data sources

• postgresql
• s3
• a custom columnar data platform (supporting Apache Flight)
• dremio
• other REST/gRPC services

Each of these have their own authorization mechanism. For example, our postgresql connections are all authenticated through Vault dynamic secrets.

I think this could be achieved by implementing our own suite of Connectors to perform the access specific specializations and then delegating to an appropriate existing Connector where it exists. I have reviewed the documentation here but I think what I really need is to internally host a suite of connectors (no one would want to reuse these as they are entirely bespoke) and then configure Dozer with those.

Does this sound feasible? Any pointers on how to get started or further reading?
Happy to provide more information as required.

[chubei]

Hi @oxlade39, thank you for your interest in Dozer!

First a note about the connector documentation page, the trait has gone through some changes. I'll update the page soon. Sorry about that.

In terms of authentication to data sources, Dozer currently only supports static credentials in some of the connectors, including postgres. Your idea of a "wrapper" Connector sounds like a cool solution to this problem, and it's totally feasible. You can use the connector to fetch a temporary credential, and create an underlying connector. After that, all method calls are forwarded to the underlying connector.

A potential pitfall is that, if the connector lost connection and tried to reconnect, the credential may have expired. And I'm not sure how postgres behaves if the role that's being used in a connection gets deleted.

Another way to work around the problem is that, instead of using a wrapper Connector, you create a "sidecar" proxy that runs along Dozer. It attaches credentials to the request that Dozer sends, gets response from the data source, and forwards the response to Dozer.

In addition, we'd also love to support dynamic credentials. If any idea comes up to you, we'll be happy to discuss!

Hope this helps!

[oxlade39]

Thanks for the reply.

The thing I really wasn't sure about is how to write my own connector, distribute that somehow and then reference via config. Is the current expectation that all connectors are distributed with dozer itself?

I hadn't thought about the sidecar approach but that does make a lot of sense. That's not something I know much about so I need to go off and do some reading.

Thanks again

[chubei]

Currently we don't have the "plugin" mechanism that allows you to write a connector and Dozer to load it dynamically. All connector code is statically built into Dozer.

Luckily it's on our roadmap. We just decided to support this in the near future. Maybe it'll be included in next release!

And you can always make your private fork of Dozer and use the static linking. It's much easier to develop and debug.