Skip to content

Commit

Permalink
feat(scaffold)!: Migrate to Arrow native SDK (#10998)
Browse files Browse the repository at this point in the history
Closes #10774

BEGIN_COMMIT_OVERRIDE
feat: Update to use [Apache Arrow](https://arrow.apache.org/) type system (#10983)

BREAKING-CHANGE: This release introduces an internal change to our type system to use [Apache Arrow](https://arrow.apache.org/). This should not have any visible breaking changes, however due to the size of the change we are introducing it under a major version bump to communicate that it might have some bugs that we weren't able to catch during our internal tests. If you encounter an issue during the upgrade, please submit a [bug report](https://github.com/cloudquery/cloudquery/issues/new/choose). You will also need to update destinations depending on which one you use:
- Azure Blob Storage >= v3.2.0
- BigQuery >= v3.0.0
- ClickHouse >= v3.1.1
- DuckDB >= v1.1.6
- Elasticsearch >= v2.0.0
- File >= v3.2.0
- Firehose >= v2.0.2
- GCS >= v3.2.0
- Gremlin >= v2.1.10
- Kafka >= v3.0.1
- Meilisearch >= v2.0.1
- Microsoft SQL Server >= v4.2.0
- MongoDB >= v2.0.1
- MySQL >= v2.0.2
- Neo4j >= v3.0.0
- PostgreSQL >= v4.2.0
- S3 >= v4.4.0
- Snowflake >= v2.1.1
- SQLite >= v2.2.0

END_COMMIT_OVERRIDE
  • Loading branch information
disq committed May 26, 2023
1 parent 4b9971e commit 69e387a
Show file tree
Hide file tree
Showing 5 changed files with 17 additions and 9 deletions.
5 changes: 3 additions & 2 deletions scaffold/cmd/templates/source/client/client.go.tpl
Expand Up @@ -3,9 +3,10 @@ package client
import (
"context"
"fmt"
"github.com/cloudquery/plugin-sdk/v2/schema"

"github.com/cloudquery/plugin-pb-go/specs"
"github.com/cloudquery/plugin-sdk/v2/plugins/source"
"github.com/cloudquery/plugin-sdk/v3/schema"
"github.com/cloudquery/plugin-sdk/v3/plugins/source"
"github.com/rs/zerolog"
)

Expand Down
5 changes: 4 additions & 1 deletion scaffold/cmd/templates/source/go.mod.tpl
Expand Up @@ -3,6 +3,9 @@ module github.com/{{.Org}}/cq-source-{{.Name}}
go 1.19

require (
github.com/cloudquery/plugin-sdk/v2 v2.7.0
github.com/cloudquery/plugin-sdk/v3 v3.6.4
github.com/rs/zerolog v1.28.0
github.com/apache/arrow/go/v13 v13.0.0-20230509040948-de6c3cd2b604
)

replace github.com/apache/arrow/go/v13 => github.com/cloudquery/arrow/go/v13 v13.0.0-20230525142029-2d32efeedad8
3 changes: 2 additions & 1 deletion scaffold/cmd/templates/source/main.go.tpl
@@ -1,8 +1,9 @@
package main

import (
"github.com/cloudquery/plugin-sdk/v2/serve"
"github.com/{{.Org}}/cq-source-{{.Name}}/plugin"

"github.com/cloudquery/plugin-sdk/v3/serve"
)

func main() {
Expand Down
5 changes: 3 additions & 2 deletions scaffold/cmd/templates/source/plugin/plugin.go.tpl
@@ -1,10 +1,11 @@
package plugin

import (
"github.com/cloudquery/plugin-sdk/v2/plugins/source"
"github.com/{{.Org}}/cq-source-{{.Name}}/client"
"github.com/cloudquery/plugin-sdk/v2/schema"
"github.com/{{.Org}}/cq-source-{{.Name}}/resources"

"github.com/cloudquery/plugin-sdk/v3/plugins/source"
"github.com/cloudquery/plugin-sdk/v3/schema"
)

var (
Expand Down
8 changes: 5 additions & 3 deletions scaffold/cmd/templates/source/resources/table.go.tpl
Expand Up @@ -3,7 +3,9 @@ package resources
import (
"context"
"fmt"
"github.com/cloudquery/plugin-sdk/v2/schema"

"github.com/apache/arrow/go/v13/arrow"
"github.com/cloudquery/plugin-sdk/v3/schema"
)

func SampleTable() *schema.Table {
Expand All @@ -13,12 +15,12 @@ func SampleTable() *schema.Table {
Columns: []schema.Column{
{
Name: "column",
Type: schema.TypeString,
Type: arrow.BinaryTypes.String,
},
},
}
}

func fetchSampleTable(ctx context.Context, meta schema.ClientMeta, parent *schema.Resource, res chan <- interface{}) error {
func fetchSampleTable(ctx context.Context, meta schema.ClientMeta, parent *schema.Resource, res chan <- any) error {
return fmt.Errorf("not implemented")
}

0 comments on commit 69e387a

Please sign in to comment.