Skip to content

data source connection should have multiple configuration #1548

@klesh

Description

@klesh

Description

This is how github connection page looks like at the moment, let's separate all fields into to 2 parts
image

  1. connection part: including the very essential settings that allow DevLake to connect to github api server
  2. configuration part: settings used for data extraction, standardization, enrichment, things of that natue

This works fine when user need to download data from a single repo on github.

However, during v0.9 patching process, we ran into following situation:

  1. We have to collect multiple repos from github into a single DevLake database
  2. Those repos share the same connection part, but may or may not share the same configuration part (repos from same org likely to share same configuration part, but not neccessary)

The workaround we chose, was to change configuration part right before we collect each repo. Obviously, this is not a long term solution that can be presented to users:

  1. Recurring pipeline (blueprint) would not work
  2. Due to reason 1, user have to manually collect multiple repos, go back and forth between connection and pipeline

Describe the solution you'd like

I don't have a clean solution at this point, so, let me list out some factors that have to be considered:

  1. configuration part is specific to the plugin, like github configuration can be completely different than jira's.
  2. configuration part may need validation
  3. configuration part may rely on connection part on usablility, like jira epic key need jira connection to fetch options for user to select.
  4. plugin may or may not need connection/configuration settings, like refdiff, it needs neither
  5. we may have to solve Multi-DataSource before this, because they could have like 1 vs Many relationship
  6. repos are connecting to configurations, do we save their relationship or accepting configuration id from options, or accepting whole configuration from options?

Has the Feature been Requested Before?

No

Metadata

Metadata

Assignees

Labels

type/feature-requestThis issue is a proposal for something new

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions