v1.2.18

Changelog

• 029e23e Feature/add go releaser to publish (slingdata-io#8)
• a5f50ca (bugfix) proton.yml: remove order by in the incremental_select (slingdata-io#25)
• 10e99b4 (bugfix): Fix inconsistent getCount() return values (slingdata-io#26)
• 4d365e7 Bugfix/disable add new col for protondb and display release version correctly (slingdata-io#9)
• aa5ebc5 Bugfix/issue 34 col name conflict (slingdata-io#35)
• 31be7b2 Feature/allow direct insert (slingdata-io#31)
• d1ca5f9 Feature/clean daily partition (slingdata-io#4)
• e369553 Feature/continue tune sling insert timeout (slingdata-io#7)
• f70b7df Feature/issue 10 switch to new version and binary naming (slingdata-io#17)
• e4fe598 Feature/release action (slingdata-io#6)
• 7c6fa50 Merge pull request #1 from yokofly/feature/support-quote-customized
• 1192eb8 Merge pull request slingdata-io#370 from slingdata-io/v1.2.19
• 787d9ab Merge pull request slingdata-io#5 from yokofly/feature/activate-insert-by-incremental-key
• 5057044 Merge remote-tracking branch 'fork/main' into protondb-enhance
• f16cd4c Merge remote-tracking branch 'fork/main' into protondb-enhance
• 6bf52f2 Protondb enhance
• 23e49c2 Protondb insert enhance(retry and idempotent support)
• edfbe4f add FileTypeRaw
• 33a213e add LASTEXITCODE
• 9d066f8 add Load Secrets (dbio)
• 604f2c7 add SLING_PROCESS_BW=false in test
• 9048a22 add SchemataLevel type
• 92601f0 add a new quote options: ✗ ./sling run --src-conn LOCAL --src-stream file://cmd/sling/tests/files/test7.csv --src-options '{ delimiter: "|", quote: "'"'"'", escape: "\\" }' --stdout 2024-09-27 02:48:17 DBG Starting pprof webserver @ localhost:6060 2024-09-27 02:48:17 DBG opened "file" connection (conn-file-dUY) 2024-09-27 02:48:17 DBG Sling version: dev (linux amd64) 2024-09-27 02:48:17 DBG type is file-file 2024-09-27 02:48:17 DBG using: {"columns":null,"mode":"full-refresh","transforms":null} 2024-09-27 02:48:17 DBG using source options: {"trim_space":false,"empty_as_null":true,"header":true,"fields_per_rec":-1,"compression":"auto","null_if":"NULL","datetime_format":"AUTO","skip_blank_lines":false,"delimiter":"|","escape":"\\","quote":"'","max_decimals":-1} 2024-09-27 02:48:17 DBG using target options: {"datetime_format":"auto","file_max_rows":0,"max_decimals":-1,"use_bulk":true,"add_new_columns":true,"adjust_column_type":false,"column_casing":"source"} 2024-09-27 02:48:17 INF reading from source file system (file) 2024-09-27 02:48:17 DBG opened "file" connection (conn-file-L9y) 2024-09-27 02:48:17 DBG reading single datastream from file://cmd/sling/tests/files/test7.csv [format=csv] 2024-09-27 02:48:17 DBG merging csv readers of 1 files [concurrency=1] from file://cmd/sling/tests/files/test7.csv 2024-09-27 02:48:17 DBG processing reader from file://cmd/sling/tests/files/test7.csv 2024-09-27 02:48:17 INF writing to target stream (stdout) col1,header,col3 data1,data with single quote,data3 data with pipe | inside,data2,data3 data with backslash \\ inside,"data with \""escaped\"" double quotes",data3 2024-09-27 02:48:17 DBG wrote 164 B: 3 rows [483 r/s] 2024-09-27 02:48:17 INF wrote 3 rows to stdout in 0 secs [476 r/s] 2024-09-27 02:48:17 INF execution succeeded ✗ cat cmd/sling/tests/files/test7.csv col1|header|col3 data1|'data with single quote \' inside'|data3 'data with pipe | inside'|data2|data3 'data with backslash \\ inside'|'data with \"escaped\" double quotes'|data3 ✗
• 17a6353 add column type testing
• fd64c58 add conn test timeout
• 731eedf add csv custom sql, improve duckdb MakeQuery
• df212fb add importViaNamedPipe duckdb
• 3ffb040 add more SLING_KEEP_TEMP
• 96645d9 add more table in proton.yaml
• 923e251 add more logging (slingdata-io#16)
• 450ac8e add more logging for drop partition (slingdata-io#14)
• f17e923 add oracle_sqlldr test, fix sqlserver test
• ad1c945 add paramKeyMapping for mysql
• dc24036 add post tests
• d706317 add purge cache
• 7b9f96b add retry for starrocks alter table
• b9d814c add skipTLSClient
• 5dea340 add skip_tls=true prop for trino
• 816ad09 add sling_cli_test
• 69a9a66 add sql mode for source parquet/iceberg/delta
• b1cdfe8 add table.MarshalJSON
• 2248709 add use_system_python for loading secrets
• cca5a09 add windows support for sqldr, format fixes
• 1f17ba1 adding another test for demo the customized quote (base) ➜ sling-cli git:(support-quote-customized) ✗ cat cmd/sling/tests/files/test8.csv col1|header|col3 data1|$data with single quote \' inside$|data3 $data with pipe | inside$|data2|data3 $data with backslash \\ inside$|$data with \"escaped\" double quotes$|data3 (base) ➜ sling-cli git:(support-quote-customized) ✗ ./sling run --src-conn LOCAL --src-stream file://cmd/sling/tests/files/test8.csv --src-options '{ delimiter: "|", quote: "$", escape: "\\" }' --stdout 2024-09-27 03:08:25 DBG Starting pprof webserver @ localhost:6060 2024-09-27 03:08:25 DBG opened "file" connection (conn-file-PhR) 2024-09-27 03:08:25 DBG Sling version: dev (linux amd64) 2024-09-27 03:08:25 DBG type is file-file 2024-09-27 03:08:25 DBG using: {"columns":null,"mode":"full-refresh","transforms":null} 2024-09-27 03:08:25 DBG using source options: {"trim_space":false,"empty_as_null":true,"header":true,"fields_per_rec":-1,"compression":"auto","null_if":"NULL","datetime_format":"AUTO","skip_blank_lines":false,"delimiter":"|","escape":"\\","quote":"$","max_decimals":-1} 2024-09-27 03:08:25 DBG using target options: {"datetime_format":"auto","file_max_rows":0,"max_decimals":-1,"use_bulk":true,"add_new_columns":true,"adjust_column_type":false,"column_casing":"source"} 2024-09-27 03:08:25 INF reading from source file system (file) 2024-09-27 03:08:25 DBG opened "file" connection (conn-file-AGU) 2024-09-27 03:08:25 DBG reading single datastream from file://cmd/sling/tests/files/test8.csv [format=csv] 2024-09-27 03:08:25 DBG merging csv readers of 1 files [concurrency=1] from file://cmd/sling/tests/files/test8.csv 2024-09-27 03:08:25 DBG processing reader from file://cmd/sling/tests/files/test8.csv 2024-09-27 03:08:25 INF writing to target stream (stdout) col1,header,col3 data1,data with single quote \' inside,data3 data with pipe | inside,data2,data3 data with backslash \\ inside,"data with \""escaped\"" double quotes",data3 2024-09-27 03:08:25 DBG wrote 174 B: 3 rows [521 r/s] 2024-09-27 03:08:25 INF wrote 3 rows to stdout in 0 secs [514 r/s] 2024-09-27 03:08:25 INF execution succeeded
• 03d35a2 adjust pattern
• 080e427 adjust the idempotent key prefix
• 035bc62 allow sqldr in windows
• b42dfd7 analyzer.go: fix typo in logging
• 0882001 change goreleaser owner
• 7c50e8a clean parition enhance: (slingdata-io#19)
• d164a5d clean up MergeDataflow
• 36673a6 clean up compile
• 9479377 config.go: set smaller batchlimit and deactivate MetadataLoadedAt by default for protondb reason: 1. protondb seems has some issue in go driver, smaller batchlimit a more stable way to send. 2. the protondb internal host a timestamp column forcely, that's why we do not need another one. _sling_loaded_at
• 6e437e9 correct print the err line
• d3efb79 database proton: switch to default tp_time type
• 4dd615f database_proton.go: enhance with retry mechanism for customized sql execution( for example we can do data mutation)
• 1b8319a database_proton.go: support insert retry
• 42f1f35 database_proton: lower trace info to err level
• d24711d delete logPath
• e36490f disable clickhouse test
• 5a1a963 disable sling update because we have no time to verify it works or not. (slingdata-io#23)
• 8e000ca exclude https from conns list
• 3ef0b7f expand replication unmarshal
• 7cf47f8 expose CleanWindowsPath
• 173c6a3 fix GetSchemata
• 8913c9a fix Upload to R2 for mac
• e7f820a fix bcp for unicode
• e47ab3c fix cli test 42
• ca2e77a fix ctlPath
• 25132f3 fix discover test
• 5ffa3a0 fix duckdb sql incremental
• 3ee4116 fix file incrmental delta/iceberg
• 9689143 fix limit in r.15
• 04c5704 fix logging typo: imoprt => import
• 5e78ad3 fix mongo primitive.ObjectID
• 8ef666e fix mysql params parsing
• 7ed23ee fix oracle getColumnsString
• 5b10671 fix oracle getColumnsString, add colNameEscaped
• 851473a fix placeholders
• d038973 fix prep.win.ps1
• 40daa9d fix replication_test
• 2b4c938 fix single file target hanging
• df0ae65 fix snowflake GenerateInsertStatement
• 58f8215 fix snowflake copy_method keys, allow GetProp for multiple keys
• b611313 fix snowflake export date parsing
• 27bea2c fix suite.cli.tsv correct use tab
• af8dee2 fix test.sh
• 966bd9a idempotent insert support
• 3830208 improve (data *Dataset) RecordsString
• 6495c5a improve ConnEntries
• 4dc6b58 improve SrcConnMD5 and TgtConnMD5
• b03c7cb improve build-dev
• 25c11f5 improve discover logging
• caa4943 improve duckdb engine
• cb9cbf6 improve mc caching for build-dev
• b389a8a improve merging for single stream
• a263858 improve panic detection
• cdadd25 improve replication ordering & tests
• 9d2bf41 improve replications makeColumns
• eba528a improve tests
• 94f15b2 improve tests
• 5f2a33e improve workflows
• 5cea304 improver performance by: 1. use batch insert api https://docs.google.com/document/d/1sdTgXVRNjNa1UemNuuFUzHWJNGQaKZcA_MiWWvJMFKw/edit#heading=h.sv54ice081a9 2. remove a sleep 1ms codeline. from: slingdata-io#337 (comment) but it introduce very pool performance: https://pastebin.ubuntu.com/p/ZwP7YDZxSg/
• 7202210 make stream_scanner entry
• 944e9b0 manual set _tp_sn type non-nullable to fix below err (slingdata-io#24)
• fa6a9ed manually set false about SLING_PROCESS_BW (slingdata-io#15)
• 3686304 move FileType to dbio, FileStreamConfig to iop
• 58ea37e next-gen duckdb
• 1533eed oracle: add infinite timeout
• c1d76fe refactor ValidateColumnNames, and GenerateInsertStatement for snowflake VARIANT inserts
• da61a4d remove TestCheck
• 3fec159 remove envkey
• cdc08c7 remove greater than or equal logic for incremental comparison
• 540cf1c remove working dir: /tmp
• ad2f4b0 set maxdecimal in proton to proton conversion (slingdata-io#32)
• 99c5c1d sling_conns.go: unlock insert into select * from. previous this query did not work well in clickhouse/protondb, because grammer.
• b95ecfb split duckdb files for build constraints
• 902ddd0 swap limit and limit_offset
• 3307502 test for success
• 49d78fa try to use csv as intermediate layout (slingdata-io#27)
• d704fc6 update (ec *EnvConns) GetConnEntry
• be03c9c update .goreleaser.windows.yaml
• fd17e9a update .goreleasers
• 20f250e update FileStreamConfig with incremental props
• a21b8b2 update GetConnEntry
• 8c8b6b0 update Prep for build
• 52b40a8 update TestSuiteDatabaseOracle
• b120e69 update build-dev
• 24495da update buld-dev
• df1fa05 update cli test 27
• d07f25e update comment
• 1fcf304 update discover local
• 9296a1f update duck.MakeScanQuery
• bc72439 update duckdb test
• bae1f1c update duckdb type map
• 4320398 update limit_sql for trino
• e183d54 update link
• fda4db3 update prep step
• 4cdd515 update prep step
• 9173961 update purge cache
• 6ee8f85 update secrets env
• 4a87e80 update sling conns Description
• 0494dbf update sling.Config for sql and data output
• 528e3a5 update sling_conns
• 6abc4a0 update snowflake private_key logic
• e524316 update sqlldr datetime casting logic
• a3562bf update suite.cli
• f8b673e update suite.cli.tsv
• 58786f0 update suite.db.template
• 19d9542 update test.sh
• 1a5704e update tests
• ebb8896 update tests for cli
• 9d33914 update type mapping logic
• 6c4194c update wildcard tests
• a1cc616 update workflows
• aee211b upgrade github.com/sijms/go-ora/v2
• 58bc06a upsert: remove deletion because protondb not support right now now incremental supported ./sling run --primary-key id --src-conn PROTON --src-stream 'v' --tgt-conn PROTON --tgt-object test_table --mode incremental
• e70890b use RemoveAllLocalTempFile
• 25b455f use RemoveLocalTempFile
• ee05671 use RemoveLocalTempFile for oracle and sqlserver
• 74a1f4a use another token instead PAT
• bd52612 use prep files
• 6d68be9 use table instead sreaming query
• 55064ad workaround to fix zero getcount return value. (slingdata-io#22)
• 115d006 working duckdb
• f567401 working motherduck