Repository navigation
Releases: yokofly/sling-cli
Releases · yokofly/sling-cli
Release list
v1.2.18-stable
Changelog
- aa07e29 refine retry
v1.2.18
Changelog
- 029e23e Feature/add go releaser to publish (slingdata-io#8)
- a5f50ca (bugfix) proton.yml: remove order by in the incremental_select (slingdata-io#25)
- 10e99b4 (bugfix): Fix inconsistent
getCount()return values (slingdata-io#26) - 4d365e7 Bugfix/disable add new col for protondb and display release version correctly (slingdata-io#9)
- aa5ebc5 Bugfix/issue 34 col name conflict (slingdata-io#35)
- 31be7b2 Feature/allow direct insert (slingdata-io#31)
- d1ca5f9 Feature/clean daily partition (slingdata-io#4)
- e369553 Feature/continue tune sling insert timeout (slingdata-io#7)
- f70b7df Feature/issue 10 switch to new version and binary naming (slingdata-io#17)
- e4fe598 Feature/release action (slingdata-io#6)
- 7c6fa50 Merge pull request #1 from yokofly/feature/support-quote-customized
- 1192eb8 Merge pull request slingdata-io#370 from slingdata-io/v1.2.19
- 787d9ab Merge pull request slingdata-io#5 from yokofly/feature/activate-insert-by-incremental-key
- 5057044 Merge remote-tracking branch 'fork/main' into protondb-enhance
- f16cd4c Merge remote-tracking branch 'fork/main' into protondb-enhance
- 6bf52f2 Protondb enhance
- 23e49c2 Protondb insert enhance(retry and idempotent support)
- edfbe4f add FileTypeRaw
- 33a213e add LASTEXITCODE
- 9d066f8 add Load Secrets (dbio)
- 604f2c7 add SLING_PROCESS_BW=false in test
- 9048a22 add SchemataLevel type
- 92601f0 add a new quote options:
✗ ./sling run --src-conn LOCAL --src-stream file://cmd/sling/tests/files/test7.csv --src-options '{ delimiter: "|", quote: "'"'"'", escape: "\\" }' --stdout 2024-09-27 02:48:17 DBG Starting pprof webserver @ localhost:6060 2024-09-27 02:48:17 DBG opened "file" connection (conn-file-dUY) 2024-09-27 02:48:17 DBG Sling version: dev (linux amd64) 2024-09-27 02:48:17 DBG type is file-file 2024-09-27 02:48:17 DBG using: {"columns":null,"mode":"full-refresh","transforms":null} 2024-09-27 02:48:17 DBG using source options: {"trim_space":false,"empty_as_null":true,"header":true,"fields_per_rec":-1,"compression":"auto","null_if":"NULL","datetime_format":"AUTO","skip_blank_lines":false,"delimiter":"|","escape":"\\","quote":"'","max_decimals":-1} 2024-09-27 02:48:17 DBG using target options: {"datetime_format":"auto","file_max_rows":0,"max_decimals":-1,"use_bulk":true,"add_new_columns":true,"adjust_column_type":false,"column_casing":"source"} 2024-09-27 02:48:17 INF reading from source file system (file) 2024-09-27 02:48:17 DBG opened "file" connection (conn-file-L9y) 2024-09-27 02:48:17 DBG reading single datastream from file://cmd/sling/tests/files/test7.csv [format=csv] 2024-09-27 02:48:17 DBG merging csv readers of 1 files [concurrency=1] from file://cmd/sling/tests/files/test7.csv 2024-09-27 02:48:17 DBG processing reader from file://cmd/sling/tests/files/test7.csv 2024-09-27 02:48:17 INF writing to target stream (stdout) col1,header,col3 data1,data with single quote,data3 data with pipe | inside,data2,data3 data with backslash \\ inside,"data with \""escaped\"" double quotes",data3 2024-09-27 02:48:17 DBG wrote 164 B: 3 rows [483 r/s] 2024-09-27 02:48:17 INF wrote 3 rows to stdout in 0 secs [476 r/s] 2024-09-27 02:48:17 INF execution succeeded ✗ cat cmd/sling/tests/files/test7.csv col1|header|col3 data1|'data with single quote \' inside'|data3 'data with pipe | inside'|data2|data3 'data with backslash \\ inside'|'data with \"escaped\" double quotes'|data3 ✗ - 17a6353 add column type testing
- fd64c58 add conn test timeout
- 731eedf add csv custom sql, improve duckdb MakeQuery
- df212fb add importViaNamedPipe duckdb
- 3ffb040 add more SLING_KEEP_TEMP
- 96645d9 add more
tablein proton.yaml - 923e251 add more logging (slingdata-io#16)
- 450ac8e add more logging for drop partition (slingdata-io#14)
- f17e923 add oracle_sqlldr test, fix sqlserver test
- ad1c945 add paramKeyMapping for mysql
- dc24036 add post tests
- d706317 add purge cache
- 7b9f96b add retry for starrocks alter table
- b9d814c add skipTLSClient
- 5dea340 add skip_tls=true prop for trino
- 816ad09 add sling_cli_test
- 69a9a66 add sql mode for source parquet/iceberg/delta
- b1cdfe8 add table.MarshalJSON
- 2248709 add use_system_python for loading secrets
- cca5a09 add windows support for sqldr, format fixes
- 1f17ba1 adding another test for demo the customized quote
(base) ➜ sling-cli git:(support-quote-customized) ✗ cat cmd/sling/tests/files/test8.csv col1|header|col3 data1|$data with single quote \' inside$|data3 $data with pipe | inside$|data2|data3 $data with backslash \\ inside$|$data with \"escaped\" double quotes$|data3 (base) ➜ sling-cli git:(support-quote-customized) ✗ ./sling run --src-conn LOCAL --src-stream file://cmd/sling/tests/files/test8.csv --src-options '{ delimiter: "|", quote: "$", escape: "\\" }' --stdout 2024-09-27 03:08:25 DBG Starting pprof webserver @ localhost:6060 2024-09-27 03:08:25 DBG opened "file" connection (conn-file-PhR) 2024-09-27 03:08:25 DBG Sling version: dev (linux amd64) 2024-09-27 03:08:25 DBG type is file-file 2024-09-27 03:08:25 DBG using: {"columns":null,"mode":"full-refresh","transforms":null} 2024-09-27 03:08:25 DBG using source options: {"trim_space":false,"empty_as_null":true,"header":true,"fields_per_rec":-1,"compression":"auto","null_if":"NULL","datetime_format":"AUTO","skip_blank_lines":false,"delimiter":"|","escape":"\\","quote":"$","max_decimals":-1} 2024-09-27 03:08:25 DBG using target options: {"datetime_format":"auto","file_max_rows":0,"max_decimals":-1,"use_bulk":true,"add_new_columns":true,"adjust_column_type":false,"column_casing":"source"} 2024-09-27 03:08:25 INF reading from source file system (file) 2024-09-27 03:08:25 DBG opened "file" connection (conn-file-AGU) 2024-09-27 03:08:25 DBG reading single datastream from file://cmd/sling/tests/files/test8.csv [format=csv] 2024-09-27 03:08:25 DBG merging csv readers of 1 files [concurrency=1] from file://cmd/sling/tests/files/test8.csv 2024-09-27 03:08:25 DBG processing reader from file://cmd/sling/tests/files/test8.csv 2024-09-27 03:08:25 INF writing to target stream (stdout) col1,header,col3 data1,data with single quote \' inside,data3 data with pipe | inside,data2,data3 data with backslash \\ inside,"data with \""escaped\"" double quotes",data3 2024-09-27 03:08:25 DBG wrote 174 B: 3 rows [521 r/s] 2024-09-27 03:08:25 INF wrote 3 rows to stdout in 0 secs [514 r/s] 2024-09-27 03:08:25 INF execution succeeded - 03d35a2 adjust pattern
- 080e427 adjust the idempotent key prefix
- 035bc62 allow sqldr in windows
- b42dfd7 analyzer.go: fix typo in logging
- 0882001 change goreleaser owner
- 7c50e8a clean parition enhance: (slingdata-io#19)
- d164a5d clean up MergeDataflow
- 36673a6 clean up compile
- 9479377 config.go: set smaller batchlimit and deactivate MetadataLoadedAt by default for protondb reason: 1. protondb seems has some issue in go driver, smaller batchlimit a more stable way to send. 2. the protondb internal host a timestamp column forcely, that's why we do not need another one. _sling_loaded_at
- 6e437e9 correct print the err line
- d3efb79 database proton: switch to default tp_time type
- 4dd615f database_proton.go: enhance with retry mechanism for customized sql execution( for example we can do data mutation)
- 1b8319a database_proton.go: support insert retry
- 42f1f35 database_proton: lower trace info to err level
- d24711d delete logPath
- e36490f disable clickhouse test
- 5a1a963 disable sling update because we have no time to verify it works or not. (slingdata-io#23)
- 8e000ca exclude https from conns list
- 3ef0b7f expand replication unmarshal
- 7cf47f8 expose CleanWindowsPath
- 173c6a3 fix GetSchemata
- 8913c9a fix Upload to R2 for mac
- e7f820a fix bcp for unicode
- e47ab3c fix cli test 42
- ca2e77a fix ctlPath
- 25132f3 fix discover test
- 5ffa3a0 fix duckdb sql incremental
- 3ee4116 fix file incrmental delta/iceberg
- 9689143 fix limit in r.15
- 04c5704 fix logging typo: imoprt => import
- 5e78ad3 fix mongo primitive.ObjectID
- 8ef666e fix mysql params parsing
- 7ed23ee fix oracle getColumnsString
- 5b10671 fix oracle getColumnsString, add colNameEscaped
- 851473a fix placeholders
- d038973 fix prep.win.ps1
- 40daa9d fix replication_test
- 2b4c938 fix single file target hanging
- df0ae65 fix snowflake GenerateInsertStatement
- 58f8215 fix snowflake copy_method keys, allow GetProp for multiple keys
- b611313 fix snowflake export date parsing
- 27bea2c fix suite.cli.tsv correct use tab
- af8dee2 fix test.sh
- 966bd9a idempotent insert support
- 3830208 improve (data *Dataset) RecordsString
- 6495c5a improve ConnEntries
- 4dc6b58 improve SrcConnMD5 and TgtConnMD5
- b03c7cb improve build-dev
- 25c11f5 improve discover logging
- caa4943 improve duckdb engine
- cb9cbf6 improve mc caching for build-dev
- b389a8a improve merging for single stream
- a263858 improve panic detection
- cdadd25 improve replication ordering & tests
- 9d2bf41 improve replications makeColumns
- eba528a improve tests
- 94f15b2 improve tests
- 5f2a33e improve workflows
- 5cea304 improver performance by: 1. use batch insert api https://docs.google.com/document/d/1sdTgXVRNjNa1UemNuuFUzHWJNGQaKZcA_MiWWvJMFKw/edit#heading=h.sv54ice081a9 2. remove a sleep 1ms codeline. from: slingdata-io#337 (comment) but it introduce very pool performance: https://pastebin.ubuntu.com/p/ZwP7YDZxSg/
- 7202210 make stream_scanner entry
- 944e9b0 manual set _tp_sn type non-nullable to fix below err (slingdata-io#24)
- fa6a9ed manually set false about SLING_PROCESS_BW (slingdata-io#15)
- 3686304 move FileType to dbio, FileStreamConfig to iop
- 58ea37e next-gen duckdb
- 1533eed oracle: add infinite timeout
- c1d76fe refactor ValidateColumnNames, and GenerateInsertStatement for snowflake VARIANT inserts
- da61a4d remove TestCheck
- 3fec159 remove envkey
- cdc08c7 remove greater than or equal logic for incremental comparison
- 540cf1c remove working dir: /tmp
- ad2f4b0 set maxdecimal in proton to proton conversion (slingdata-io#32)
- 99c5c1d sling_conns.go: unlock
insert into select * from. previous this query did not work well in clickhouse/protondb, because grammer. - b95ecfb split duckdb files for build constraints
- 902ddd0 swap limit and lim...
v3.5.6-rc1
v2.1.12
v2.1.11
v2.0.3-rc1
Changelog
- 029e23e Feature/add go releaser to publish (slingdata-io#8)
- 4d365e7 Bugfix/disable add new col for protondb and display release version correctly (slingdata-io#9)
- d1ca5f9 Feature/clean daily partition (slingdata-io#4)
- e369553 Feature/continue tune sling insert timeout (slingdata-io#7)
- e4fe598 Feature/release action (slingdata-io#6)
- 7c6fa50 Merge pull request #1 from yokofly/feature/support-quote-customized
- 1192eb8 Merge pull request slingdata-io#370 from slingdata-io/v1.2.19
- 787d9ab Merge pull request slingdata-io#5 from yokofly/feature/activate-insert-by-incremental-key
- 5057044 Merge remote-tracking branch 'fork/main' into protondb-enhance
- f16cd4c Merge remote-tracking branch 'fork/main' into protondb-enhance
- 6bf52f2 Protondb enhance
- 23e49c2 Protondb insert enhance(retry and idempotent support)
- edfbe4f add FileTypeRaw
- 33a213e add LASTEXITCODE
- 9d066f8 add Load Secrets (dbio)
- 604f2c7 add SLING_PROCESS_BW=false in test
- 9048a22 add SchemataLevel type
- 92601f0 add a new quote options:
✗ ./sling run --src-conn LOCAL --src-stream file://cmd/sling/tests/files/test7.csv --src-options '{ delimiter: "|", quote: "'"'"'", escape: "\\" }' --stdout 2024-09-27 02:48:17 DBG Starting pprof webserver @ localhost:6060 2024-09-27 02:48:17 DBG opened "file" connection (conn-file-dUY) 2024-09-27 02:48:17 DBG Sling version: dev (linux amd64) 2024-09-27 02:48:17 DBG type is file-file 2024-09-27 02:48:17 DBG using: {"columns":null,"mode":"full-refresh","transforms":null} 2024-09-27 02:48:17 DBG using source options: {"trim_space":false,"empty_as_null":true,"header":true,"fields_per_rec":-1,"compression":"auto","null_if":"NULL","datetime_format":"AUTO","skip_blank_lines":false,"delimiter":"|","escape":"\\","quote":"'","max_decimals":-1} 2024-09-27 02:48:17 DBG using target options: {"datetime_format":"auto","file_max_rows":0,"max_decimals":-1,"use_bulk":true,"add_new_columns":true,"adjust_column_type":false,"column_casing":"source"} 2024-09-27 02:48:17 INF reading from source file system (file) 2024-09-27 02:48:17 DBG opened "file" connection (conn-file-L9y) 2024-09-27 02:48:17 DBG reading single datastream from file://cmd/sling/tests/files/test7.csv [format=csv] 2024-09-27 02:48:17 DBG merging csv readers of 1 files [concurrency=1] from file://cmd/sling/tests/files/test7.csv 2024-09-27 02:48:17 DBG processing reader from file://cmd/sling/tests/files/test7.csv 2024-09-27 02:48:17 INF writing to target stream (stdout) col1,header,col3 data1,data with single quote,data3 data with pipe | inside,data2,data3 data with backslash \\ inside,"data with \""escaped\"" double quotes",data3 2024-09-27 02:48:17 DBG wrote 164 B: 3 rows [483 r/s] 2024-09-27 02:48:17 INF wrote 3 rows to stdout in 0 secs [476 r/s] 2024-09-27 02:48:17 INF execution succeeded ✗ cat cmd/sling/tests/files/test7.csv col1|header|col3 data1|'data with single quote \' inside'|data3 'data with pipe | inside'|data2|data3 'data with backslash \\ inside'|'data with \"escaped\" double quotes'|data3 ✗ - 17a6353 add column type testing
- fd64c58 add conn test timeout
- 731eedf add csv custom sql, improve duckdb MakeQuery
- df212fb add importViaNamedPipe duckdb
- 3ffb040 add more SLING_KEEP_TEMP
- 96645d9 add more
tablein proton.yaml - 923e251 add more logging (slingdata-io#16)
- 450ac8e add more logging for drop partition (slingdata-io#14)
- f17e923 add oracle_sqlldr test, fix sqlserver test
- ad1c945 add paramKeyMapping for mysql
- dc24036 add post tests
- d706317 add purge cache
- 7b9f96b add retry for starrocks alter table
- b9d814c add skipTLSClient
- 5dea340 add skip_tls=true prop for trino
- 816ad09 add sling_cli_test
- 69a9a66 add sql mode for source parquet/iceberg/delta
- b1cdfe8 add table.MarshalJSON
- 2248709 add use_system_python for loading secrets
- cca5a09 add windows support for sqldr, format fixes
- 1f17ba1 adding another test for demo the customized quote
(base) ➜ sling-cli git:(support-quote-customized) ✗ cat cmd/sling/tests/files/test8.csv col1|header|col3 data1|$data with single quote \' inside$|data3 $data with pipe | inside$|data2|data3 $data with backslash \\ inside$|$data with \"escaped\" double quotes$|data3 (base) ➜ sling-cli git:(support-quote-customized) ✗ ./sling run --src-conn LOCAL --src-stream file://cmd/sling/tests/files/test8.csv --src-options '{ delimiter: "|", quote: "$", escape: "\\" }' --stdout 2024-09-27 03:08:25 DBG Starting pprof webserver @ localhost:6060 2024-09-27 03:08:25 DBG opened "file" connection (conn-file-PhR) 2024-09-27 03:08:25 DBG Sling version: dev (linux amd64) 2024-09-27 03:08:25 DBG type is file-file 2024-09-27 03:08:25 DBG using: {"columns":null,"mode":"full-refresh","transforms":null} 2024-09-27 03:08:25 DBG using source options: {"trim_space":false,"empty_as_null":true,"header":true,"fields_per_rec":-1,"compression":"auto","null_if":"NULL","datetime_format":"AUTO","skip_blank_lines":false,"delimiter":"|","escape":"\\","quote":"$","max_decimals":-1} 2024-09-27 03:08:25 DBG using target options: {"datetime_format":"auto","file_max_rows":0,"max_decimals":-1,"use_bulk":true,"add_new_columns":true,"adjust_column_type":false,"column_casing":"source"} 2024-09-27 03:08:25 INF reading from source file system (file) 2024-09-27 03:08:25 DBG opened "file" connection (conn-file-AGU) 2024-09-27 03:08:25 DBG reading single datastream from file://cmd/sling/tests/files/test8.csv [format=csv] 2024-09-27 03:08:25 DBG merging csv readers of 1 files [concurrency=1] from file://cmd/sling/tests/files/test8.csv 2024-09-27 03:08:25 DBG processing reader from file://cmd/sling/tests/files/test8.csv 2024-09-27 03:08:25 INF writing to target stream (stdout) col1,header,col3 data1,data with single quote \' inside,data3 data with pipe | inside,data2,data3 data with backslash \\ inside,"data with \""escaped\"" double quotes",data3 2024-09-27 03:08:25 DBG wrote 174 B: 3 rows [521 r/s] 2024-09-27 03:08:25 INF wrote 3 rows to stdout in 0 secs [514 r/s] 2024-09-27 03:08:25 INF execution succeeded - 03d35a2 adjust pattern
- 080e427 adjust the idempotent key prefix
- 035bc62 allow sqldr in windows
- b42dfd7 analyzer.go: fix typo in logging
- 764b621 change version: for release canndidate: binary: sling version: 1.2.20-rc{1,2,3...}-timeplus
- 4560978 clean logging, follow up for slingdata-io#11
- d164a5d clean up MergeDataflow
- 36673a6 clean up compile
- 9479377 config.go: set smaller batchlimit and deactivate MetadataLoadedAt by default for protondb reason: 1. protondb seems has some issue in go driver, smaller batchlimit a more stable way to send. 2. the protondb internal host a timestamp column forcely, that's why we do not need another one. _sling_loaded_at
- 535e727 continue disable SLING_PROCESS_BW if we use source as timeplus database.
- 6e437e9 correct print the err line
- d3efb79 database proton: switch to default tp_time type
- 4dd615f database_proton.go: enhance with retry mechanism for customized sql execution( for example we can do data mutation)
- 1b8319a database_proton.go: support insert retry
- 42f1f35 database_proton: lower trace info to err level
- d24711d delete logPath
- e36490f disable clickhouse test
- 8e000ca exclude https from conns list
- 3ef0b7f expand replication unmarshal
- 7cf47f8 expose CleanWindowsPath
- 173c6a3 fix GetSchemata
- 8913c9a fix Upload to R2 for mac
- e7f820a fix bcp for unicode
- e47ab3c fix cli test 42
- ca2e77a fix ctlPath
- 25132f3 fix discover test
- 5ffa3a0 fix duckdb sql incremental
- 3ee4116 fix file incrmental delta/iceberg
- 9689143 fix limit in r.15
- 04c5704 fix logging typo: imoprt => import
- 5e78ad3 fix mongo primitive.ObjectID
- 8ef666e fix mysql params parsing
- 7ed23ee fix oracle getColumnsString
- 5b10671 fix oracle getColumnsString, add colNameEscaped
- 851473a fix placeholders
- d038973 fix prep.win.ps1
- 40daa9d fix replication_test
- 2b4c938 fix single file target hanging
- df0ae65 fix snowflake GenerateInsertStatement
- 58f8215 fix snowflake copy_method keys, allow GetProp for multiple keys
- b611313 fix snowflake export date parsing
- 27bea2c fix suite.cli.tsv correct use tab
- af8dee2 fix test.sh
- 966bd9a idempotent insert support
- 3830208 improve (data *Dataset) RecordsString
- 6495c5a improve ConnEntries
- 4dc6b58 improve SrcConnMD5 and TgtConnMD5
- b03c7cb improve build-dev
- 25c11f5 improve discover logging
- caa4943 improve duckdb engine
- cb9cbf6 improve mc caching for build-dev
- b389a8a improve merging for single stream
- a263858 improve panic detection
- cdadd25 improve replication ordering & tests
- 9d2bf41 improve replications makeColumns
- eba528a improve tests
- 94f15b2 improve tests
- 5f2a33e improve workflows
- 5cea304 improver performance by: 1. use batch insert api https://docs.google.com/document/d/1sdTgXVRNjNa1UemNuuFUzHWJNGQaKZcA_MiWWvJMFKw/edit#heading=h.sv54ice081a9 2. remove a sleep 1ms codeline. from: slingdata-io#337 (comment) but it introduce very pool performance: https://pastebin.ubuntu.com/p/ZwP7YDZxSg/
- 7202210 make stream_scanner entry
- fa6a9ed manually set false about SLING_PROCESS_BW (slingdata-io#15)
- 3686304 move FileType to dbio, FileStreamConfig to iop
- 58ea37e next-gen duckdb
- 1533eed oracle: add infinite timeout
- c1d76fe refactor ValidateColumnNames, and GenerateInsertStatement for snowflake VARIANT inserts
- da61a4d remove TestCheck
- 3fec159 remove envkey
- cdc08c7 remove greater than or equal logic for incremental comparison
- 540cf1c remove working dir: /tmp
- 99c5c1d sling_conns.go: unlock
insert into select * from. previous this query did not work well in clickhouse/protondb, because grammer. - b95ecfb split duckdb files for build constraints
- 902ddd0 swap limit and limit_offset
- 3307502 test for success
- d704fc6 update (ec *EnvConns) GetConnEntry
- be03c9c update .goreleaser.windows.yaml
- fd17e9a update .goreleasers
- 20f250e update FileStreamConfig with incremental props
- a21b8b2 update GetConnEntry
- 8c8b6b0 update Prep for build
- 52b40a8 update TestSuiteDatabaseOracle
- b120e69 update build-dev
- 24495da update buld-dev
- df1fa05 upda...
v1.3.4-rc1
v2.1.9
Changelog
- 979a6b1 Update release.yml
v2.0.12
Changelog
- 979a6b1 Update release.yml
v2.0.11
Changelog
- 0ada4f5 Update release.yml