Development Release - dev

Bug Fixes

• Handled groupby with no dimensions fields (#68)
• first flag in CacheIterator was not set correctly (#101)
• CacheIterator descending from past db end (#105)
• Fix cache changes on update operation (#116)
• Benches report missing the comparison (#183)
• Fix schema validation when only some tables are used (#174)
• Wait until all schema initialized before start grpc server (#187)
• Secondary index key should never be empty (#200)
• Make dynamic server optional
• Redirect events to the right endpoint (#206)
• Db settings + schema checkpointing (#227)
• Remove all features flag in coverage (#255)
• Synchronize termination among multiple sources (#257)
• Remove array types (#259)
• Fix file path for snowflake benchmark (#297)
• Dont define snowflake as default feature for dozer-admin (#304)
• Rely on cache instead of events (#308)
• Use badge from main branch (#327)
• Simplify Aggregator test (#338)
• Update serial_test to 0.9.0 because the old version depended (#344)
• Use source based coverage and ignore ../* (#347)
• Reduce schema update calls for snowflake ingestion (#358)
• Numbers were serialized as strings in REST API (#369)
• Fix config folder location for rustdoc (#372)
• Allow conversion from null when field is nullable (#371)
• Seconary index was not calculated properly during querying (#366)
• Add param experimental_allow_proto3_optional as default when compiling protoc (#342)
• Implement proper timeout handling in sql tests (#378)
• Single order_by query didn't work (#375)
• order_by without range filter didn't work (#382)
• Handle NULL in cache (#391)
• Support for NULL and Decimal in SUM() aggregator (#394)
• Retrieve old record from db in sqlite tests (#392)
• Criterion build, remove build dependency (#356)
• Fix bench functions (#403)
• Implement NULL support for SQL expressions (#405)
• Reuse connection (#412)
• Update changes structure (#415)
• Fix coverage tests configs (#416)
• Remove conflicting parameters from tests sources config (#417)
• Bring back notifier which was wrongly removed in getdozer#407 (#424)
• Display error of dozer app (#429)
• Revert replication slots check query changes (#433)
• Readonly LmdbCache didn't open secondary index databases (#439)
• Handle eth data conversion for longer strings and fix a few orchestrator errors (#446)
• Handle event name conflicts (#449)
• Fix RDS user validation for postgres replication (#454)
• Cache benchmark was panicking because duplicated insertion (#461)
• Unwrap errors from pipeline (#460)
• Synchronize commits across multiple sources (#471)
• Fix schema initiation and port mapping in new source builder (#486)
• Hanging dag tests (#494)
• Increase max readers on lmdb cache (#497)
• Improve errors (#498)
• Readme (#502)
• Broken link in readme (#504)
• Synchronize sources startup (to pre-populate RecordReader) (#511)
• Propagate errors from connector threads. (#517)
• Oapi schema was not generated correctly (#545)
• Propagate nullables along SQL schema (#546)
• Update samples with public ecr image (#551)
• Fix postgres replication messages handling on update (#550)
• Id in LmdbCache can be duplicated for different records (#568)
• Remove duplicate dependencies and bump a few libs. (#571)
• Display user friendly postgres connection failure (#573)
• dozer-tests could not be started due to last renaming refactor (#579)
• Generate descriptor once per init (#591)
• Handle error return from thread when intialize grpc + rest (#581)
• Typos and links in proto files (#593)
• Fix eth e2e test. Update primary key mismatch error message. (#598)
• Examples and Readme (#617)
• Count APIs should default to no limit (#623)
• Discord notification blocked because PR is from public fork (#630)
• Dozer-api test utils was generating inconsistent schema and record (#635)
• Count API still used default limit when query string is empty (#634)
• Schema id was not copied to records before inserting (#636)
• Allow text fields being used in full text index (#639)
• Generate full text index for String fields (#638)
• Ignore unused sources for pipeline (#646)
• Copy schema id to records before letting cache handle it (#649)
• Fix sql parsing panic and statements being silently ignored (#642)
• Need ignore_pipeline_branch_filters to trigger when pushes are disabled (#662)
• OnEvent is not propagating data in some scenarios with postgres (#637)
• Don't pass commit message to BuildKite. (#665)
• Update issue template config with parent item (#677)
• Change mapping of sqlite data type to dozer data type (#689)
• CONCAT() to support multiple arguments (#690)
• Unify typed service query and on_event record representation (#647)
• Implement record versioning in RecordWriter and RecordReader (#682)
• Fix minor snowflake connector issues (#681)
• Print unknown command error and continue (#671)
• Forbid duplicated cte names (#703)
• Use context to solve alias properly (#708)
• Fix snowflake connector (#745)
• Unable clean untagged images for now (#752)
• Fix source with empty columns list validation (#714)
• serde_str is needed to bincode deserialize Decimal (#757)
• Don't commit when no source produces new op (#669)
• Fix stream exist check for tables (#754)
• Architecture diagram (#758)
• Fix snowflake stream consumption (#765)
• ApiEndpoint is not serializing all fields (#776)
• Ethereum end to end test (#779)
• Integration tests required dozer-tests image when using local runner (#778)
• Clear PK from Projection output (#780)
• Fix record delete when duplicated words are in string field (#800)
• Fix usage of decimal with trailing zeros (#810)
• Fix e2e tests and hanging epoch manager test (#812)
• Rephrase error when primary key is missing (#813)
• Buildkite integration test for eth e2e case (#841)
• Datafusion connector compilation (#891)
• TypedService didn't filter events based on endpoint (#899)
• Fix connector ls command of object storage connector (#884)
• Use correct path builder for local storage (#924)
• Buildkite integration test didn't get ETH_HTTPS_URL env var (#945)
• Fix eth-traces e2e test and temporarily remove eth_union e2e test (#952)
• Health check should not return NotServing when dynamic is false (#923)
• Config in e2e test flags_dynamic_false (#953)
• Fix discord notifications. (#967)
• Variable usage in discord (#975)
• Fix release.yml identation (#983)
• Add epsilon to distance test case (#981)
• Don't create default connector config (#964)
• Discord notification on push to main (#998)
• Remove leaked in eth_union e2e case (#997)
• Eth blocks parse fails (#1004)
• Invalid column error (#1008)
• Remove schema unwrap (#1010)
• Fix snowflake table name validation (#1011)
• Fix snowflake schema helepr import (#1014)
• Fix replication slot check (#1031)
• Remove non-existent proto include, which made cargo always rerun (#1064)
• Record readers not inserted in release builds (#1068)
• timestamp, point, decimal support for grpc (#1060)
• Select * wildcard (#1080)
• Fix convert from jsonb to dozer type (#1086)
• Fix check of supported tables in postgres (#1075)
• Clean dozer on start because blue/green cache is not switched correctly (#1091)
• Fix order of columns in schema (#1090)
• Primary key field conversion from REST url (#1097)
• Sort schemas by sources order from dozer config (#1095)
• Use same progress bar style (expect for color) for source and sink (#1100)
• Updated download_binary (#1107)
• Sink should only be built after all source checkpoints are checked (#1112)
• Blue cache was counted with limit. Green cache didn't get picked up. (#1123)
• Define unique temp folder for aggregation tests (#1124)
• Groupby without measures were handled wrongly (#1127)
• Add postgres columns validations and connector test (#1035)
• Ingestion via Arrow (#1141)
• load_multi_sources test was not implemented correctly (#1146)
• Remove table name property usage in deltalake connector (#1155)
• Fix compilation error introduced in getdozer#1158 (#1183)
• Add sources names used in the query (#1190)
• PK was not automatically when GROUP BY is used (#1193)
• Fix publication slot creation (#1202)
• Lmdb environments should use the same map size (#1216)
• Restore Docker build (#1218)
• Broken e2e tests by various PRs (#1235)
• Fix postgres validation with schemas usage (#1243)
• Don't panic if operation cannot be read from indexing thread. (#1247)
• Broken link (#1268)
• arrow::Inverval is not supported in data conversion but supported in type converion (#1295)
• Derived builder (#1297)
• Fix data conversion issues in local storage (#1301)
• Use cdc type "nothing" when no PK defined (#1307)
• Postgres BIT is supported in type conversion but not value conversion (#1305)
• Fix error handle in processor (#1325)
• Division by zero error handling (#1324)
• Update .deb url doc (#1333)
• Multimap single item removal (#1336)
• Api should use read only cache manager (#1346)
• Switch join order for empty cases (#1348)
• Openssl dependecies as features (#1358)
• README Ubuntu instructions (#1367)
• Fix web config usage in auth grpc service (#1380)
• Minor error while running with duration (#1388)
• Update resolution atomicity and stream operations. (#1390)
• Grpc error caused by duration (#1395)
• E2e tests write to directories that are not removed (#1407)
• Type validation (#1405)
• Use tokio::time::sleep instead of std::thread::sleep (#1413)
• Api thread graceful shutdown (#1416)
• Use proper error message for postgres replication slot creation (#1422)
• Use rustls reqwest (#1443)
• Uuid postgres type support (#1458)
• Add Record missing fields (#1525)
• OpenTelemetry must be initialized in a tokio runtime (#1541)
• OpenTelemetry tracing not working because of Jaeger batch exporter. (#1549)
• Update style of progress bar (#1567)
• Map grpc call error to proper error (#1570)
• Numeric (10, 2) (#1583)
• Tests behind the feature flag "mongodb" didn't run on ci (#1593)
• api_desired and api_available should be optional (#1602)
• Fix file names sent to cloud (#1607)
• Log client didn't authenticate itself (#1620)
• Include progress printer only when cloud feature enabled (#1635)
• ApiEndpoint::conflict_resolution and ApiEndpoint::version was not serialized (#1643)
• Get a deterministic Processor id (#1665)
• Index in ui graph (#1671)
• Use rustls for tls support when connecting to postgres (#1502)
• Fix date conversion when date is more precise (#1682)
• Append only (#1673)
• Avoid include snapshot event in metrics count (#1686)
• Remove secrets value and add secrets support in deploy command (#1697)
• Show error when app id is not stored (#1699)
• Collide metric key, change to cache_operation_log for OperationLog feat (#1687)
• Ignore a failing test. Fix is in progress. (#1709)
• Write commit epoch to log (#1711)
• Handle rest path collision (#1693)
• Properly log error when internal pipeline server can't be started (#1715)
• --config-overrides was not exposed in cli (#1743)
• Use different slot name everytime when dozer connects to postgres (#1742)
• Dont serialize app name if it is missing (#1746)
• Log reader was duplicating records that were persisted (#1750)
• Fix config generation and log output in init command (#1754)
• Replication log could panic if received request during persisting (#1759)
• Delete config file after delete of app (#1766)
• Add missing flags and comments in list and logs commands (#1767)
• Show warning when metadata collect failed (#1772)
• Fix config combination with separated sql scripts (#1773)
• Fix target url from cli args usage (#1774)
• Truncate credentials file when new profiles added or updated (#1783)
• Fix display of missing app error (#1784)
• From<Record> of ProcessorRecord didn't copy lifetime (#1789)
• Temporary Remove python log s390x build (#1807)
• Custom enum row detection, support different type Coalesce (#1802)
• Fix inconsistent prost enum tag annotation (#1817)
• Don't hardcode s3 storage region (#1822)
• Should use tokio mutex in async context (#1842)
• Support uploading empty multipart file to s3 (#1846)
• Mark log creation to avoid writing to old log version (#1850)
• Don't start api until internal server is up (#1861)
• Should not panic if source sender errors out because of channel error (#1864)
• Start dozer live ui server (#1874)

Chore

• Add build folder to ignored list for coveralls (#296)

Documentation

• Add build dependences in README (#96)
• Update postgres requirements (#307)
• Include kill_cache=1 param (#370)
• Add gRPC proto file documentation (#477)
• Include contributing section (#609)
• Update logo (#633)
• Init connector README (#872)
• Add type conversion spec in API (#1313)
• Add react sample URL reference (#1491)
• Update sample table & other references (#1652)

Features

• Init ingestion postgres
• Reorganise into a mono repo (#6)
• Merge pipeline code (#7)
• Merge api (#8)
• Implement first version of storage (#17)
• Refactor connector * iterator behavior (#24)
• Update issue templates
• Implement type conversion for postgresql (#25)
• Implement basic cache operations (#48)
• Implement secondary indexes (#49)
• Implement query bench (#50)
• Implement proper schema id usage in snapshotter and refactor mapping (#51)
• Implement simple orchestrator (#61)
• Init dozer ci (#77)
• Include concurrency and multiple restore keys (#82)
• Implement multi ingestion sources (#78)
• Implement continue of replication messages stream handling (#94)
• Dozer-api REST (#86)
• Allows to specify state store options (#112)
• Add FullText IndexType (#121)
• Multi-key indexer for state store (#127)
• Implement e2e tests (#130)
• Supports for transactions across multiple threads (#132)
• Implement auth module and middleware (#109)
• Implement the Aggregation Builder (#129)
• Remove schema registry (#138)
• Implement plan generation (#149)
• Write github action for Long running tests, Performance benchmarks, Code vulnerability, Code coverage etc (#146)
• Add import google libs when generate proto with special type (#165)
• Implement schema validator for postgresql (#128)
• Implement indexed record store (#181)
• Implement eth connector (#178)
• Implement cursor modify operations (#182)
• Introduce generic GRPC queries (#194)
• Support descending fields in sorted inverted index (#202)
• Implement release process (#204)
• Shareable transactions (#186)
• Validate postgres connection details and replication availability (#143)
• Implement stateful sources (#219)
• Improve cli and pipeline experience. (#226)
• Implement time threshold for checkpointing (#228)
• UCASE() support for scalar function (#218)
• Implement batched writer for cache (#230)
• Integrate odbc connector for snowflake (#191)
• Init test framework (#247)
• Implement checkpointing v1 (#254)
• Associate a unique integer id to every record. (#232)
• Use Iterator in cache query (#264)
• Product Processor (#220)
• Notify processors and sinks before commit (#266)
• Implement prefix-aware transaction (#271)
• Implement debezium connector (#246)
• Implement build script for dozer-admin (#295)
• Support query using multiple index scans (#314)
• Support selective columns usage of source (#323)
• Support for key/value based aggregators (#340)
• AVG/MIN/MAX support for Float and Int (#341)
• Support filter in grpc event stream (#334)
• Decimal support for Aggregators (#348)
• Timestamp & Date support in MIN/MAX aggregator (#361)
• Implement schema mapping for snowflake (#364)
• Migrate to static schema
• Implement get_schema for debezium (#374)
• Implement Application to facilitate the binding of sources + pipelines (#383)
• Replace seqno with txid/seq_in_tx to support checkpoint recovery (#389)
• Resume execution for check-pointed state store (#390)
• Implement schema registry support for debezium (#386)
• Notify source_id, txid, seq_in_tx to processors and sinks (for cache commit propagation) (#400)
• Implement tracing (#351)
• Implement cli experience for dozer admin (#368)
• Implement validation for tables and columns names (#425)
• Add home_dir configuration in dozer config + api + pipeline + cache (#413)
• Parse abi and events and publish as tables (#440)
• Move streaming to pipeline server (#414)
• Pretty print schema (#452)
• Implement benchmarking for postgres (#458)
• Grpc authentication layer (#451)
• Implement eth sample and tests (#470)
• Add prepare() method in Dag (#483)
• Implement init cli command (#482)
• Make push events optional (#493)
• /health endpoint for rest and grpc (#495)
• Implement RecordWriter supporting automatic rowkey generation (#530)
• Support join operator (#476)
• Implement TRIM() (#540)
• Support for field aliases in SQL (#552)
• Implement interactive REPL for viewing sources and testing data (#558)
• Map replication types with output port types (#547)
• Implement continue for postgres ingestion (#515)
• Support LIKE operator (#582)
• Add count API (#588)
• Add validation step to init command (#578)
• Support CAST operator (#592)
• Support UInt in aggregation (#654)
• Implement ingestion continue for eth and fix minor issues (#661)
• Add and use table name from table info struct (#670)
• Implement nested queries and CTE. (#680)
• Support bpchar column type (#687)
• Atomatically trim record history in RecordWriter (#699)
• Add sql_date support to snowflake integration (#721)
• Implement direct insert to cache pipeline (#744)
• Select wildcard (#746)
• Implement stream creation on views in snowflake (#749)
• Change app config to support source name in api endpoints (#762)
• Implement multi-way JOIN (#729)
• Adding commit SHA for ECR image tag for release (#787)
• Adding commit SHA for ECR image tag for release (#789)
• Implement basic continue for ingestion in snowflake (#775)
• Support LEFT/RIGHT JOIN (#788)
• All nodes use the same instant for commit timeout (#807)
• Refactor sql to be common across sources (#803)
• Stateful pipeline for sources without PK (#817)
• Introduce DataFusion Connector (#808)
• Return record id in common grpc query (#879)
• Add record version to push and pull query results (#896)
• Return id and version in gRPC query. Include version in push events. (#900)
• Return record id and version in REST API (#898)
• Support $after expression for pagination (#908)
• LmdbCacheManager and endpoint as an alias (#919)
• Generate graph (#911)
• Integrate dashboard (#917)
• Support python udf (#927)
• Add geo point and coord type support (#912)
• Include id in grpc event stream (#963)
• Expose Pipeline config (#946)
• Math support for UInt (#933)
• Set operator lmdb support (#961)
• decimal and binary support for grpc (#1015)
• Parse yaml (#996)
• Implement start from in postgres connector (#994)
• Add RoCacheEndpoint::redirect_cache. (#1027)
• Enhance python udf (#984)
• Support timestamp diff (#1074)
• Blue green cache (#1061)
• DAY_OF_WEEK() + SQL extensive type support for operations (#1076)
• Implement progress for sources (#1079)
• Read data from postgres snapshot in parallel (#1094)
• Switch to new cache if it processed enought number of operations (#1092)
• Implement arrow format for grpc ingestion (#1087)
• Add LmdbMap (#1151)
• Add LmdbMultimap (#1153)
• Support deltalake connector (#1067)
• SET operators record_map into CountingBloomFilter (#1168)
• Window processor (#1175)
• Add operation log to cache (never cleaned up now) (#1172)
• Implement date extract (#1178)
• Implement tracing using open telemetry (#1176)
• Separate cache operation log environment and index environments (#1199)
• Asynchoronous indexing (#1206)
• Implement schema support for postgres (#1070)
• Add object store validation (#1140)
• Resolve sink and processor errors (#1267)
• Handle grpc API errors gracefully (#1289)
• Make sources optional (#1318)
• COALESCE() support (#1342)
• Implement ExecutionLog on fs 🚀 (#1356)
• Implement authorization in gRPC methods (#1375)
• U128 and i128 Field and FieldType (#1315)
• Add snapshotting started message to ingestion process (#1377)
• Read object store files in parallel (#1381)
• Duration Field & FieldType support (#1374)
• Duration for sql representation + aggregation support (#1384)
• SQL Logic Test Framework (#1326)
• Implement APIs for monitoring and status updates (#1329)
• Use record hash as key when schema has no primary index (#1399)
• Implement js log bindings (#1424)
• Implement python log bindings (#1426)
• Support loading dozer config from an url. (#1440)
• Switch to postgres for admin database (#1444)
• dozer deploy command (#1435)
• Make secondary index configurable (#1461)
• Allow disabling of grpc & rest apis via config (#1441)
• Now() (#1467)
• Add support for cloud list and status methods (#1507)
• json & jsonb and their array type support (#1200)
• Last version check (#1518)
• Count(*) sql support (#1520)
• Send files list with contents in cloud deploy call (#1523)
• Add Record lifetime (#1453)
• Run migration only when necessary (#1515)
• Allow starting API on selected migration number (#1530)
• Add expression support in TTL (#1529)
• JSON_VALUE() & JSON_QUERY() sql support (#1526)
• Api server can continue cache building from checkpoint (#1538)
• Add update and delete for cloud APIs (#1544)
• Add metrics for cache (#1551)
• Add public service provide metadata for cli - dozer cloud login (#1553)
• Implement monitoring command (#1559)
• Display progress of cloud deployment (#1566)
• Add stop app in cloud call (#1572)
• Add filters and pagination to command (#1573)
• Show revision and version information in cloud status (#1574)
• Support version management (#1575)
• Update cloud proto to create connection and source manually (#1584)
• Add phase endpoint (#1585)
• Return endpoint paths at REST API root (#1587)
• Add cloud configuration to Dozer config (#1586)
• Add dozer cloud version status {version} command (#1588)
• Update LogMessageRequest and LogMessage to include follow option and the log source (#1590)
• Create empty directories for lambdas and queries during init (#1591)
• Add cloud deploy -n and dozer cloud update -n (#1597)
• Add cloud api set-num-replicas (#1598)
• Implement error threshold for processor and sink errors (#1596)
• Add deployment number to get status proto file (#1600)
• Implement logs -f and logs -d deployment (#1605)
• Dozer cloud login (#1604)
• Emit metrics of source operations and pipeline latency (#1610)
• Emit Product processor metrics (#1611)
• Refactor kafka connector to use schema registry (#1622)
• Add more cache metrics (#1627)
• Implement progress printer and disable time display (#1630)
• Implement cache dump and restore (#1637)
• Switch to rdkafka (#1636)
• Add api latency metric (#1642)
• Allow use kafka without schema registry (#1626)
• _text (#1645)
• Add data latency metrics. (#1650)
• CASE statement (#1656)
• Support generate UI graph from dozer config (#1661)
• Implement append-last watcher (#1562)
• Pg tls config connectivity (#1672)
• Add cloud secrets API interfaces to cloud.proto (#1689)
• Qualified wildcard (#1674)
• Store app id to context file (#1691)
• Implement secrets commands (#1695)
• Implement log replication server core (#1710)
• Support json files for cloud deployments (#1714)
• Support InList clause in streaming SQL (#1694)
• Support read of configs from multiple files (#1718)
• Show deployment logs from deploy API (#1731)
• Implement config override (#1736)
• MAX_VALUE(expr1, expr2) (#1733)
• Serve replication log on internal grpc server (#1722)
• MIN_VALUE() implementation + unit tests (#1740)
• Show correct error when organisation doesnt exist (#1756)
• Use config argument in deploy command (#1758)
• Emit sql graph (#1765)
• Add new label snapshotting to cache_operation metric (#1737)
• Register errors from error manager in jaeger (#1753)
• Refactor records (#1781)
• Marker file detection for object storage (#1790)
• Allow to use client properties in login command (#1813)
• Synchronize GrpcConfig with proto and add phase in AppResponse (#1830)
• Implement live code reload experience. (#1826)
• Udf onnx config placeholder (#1831)
• Add list_endpoints to internal server (#1823)
• Publish DAG to JSON (#1824)
• Dowload and use dozer-live ui (#1848)
• Replace jaeger with xray (#1862)
• Implement graph on live ui (#1847)
• Introduce MySQL Connector (#1829)

Fix

• Add back import google lib types depend on schema (#313)
• Fix cache benchmarking (#1294)

Miscellaneous Tasks

• Include lock file
• Refactor name and minor changes (#16)
• Convert REST to GRPC (#20)
• Fix warnings (#63)
• Fix schema insert and warnings (#72)
• Remove sql scripts (#76)
• Run on every branch
• Use slices instead (#79)
• Fix dozer.yaml (#84)
• Restore branch after refactor (#81)
• Switch error management to anyhow (#85)
• Fix clippy warnings (#88)
• Add log4rs file + logging for processors (#90)
• Improve ci build time (#89)
• Generate schema_id in Sink (#98)
• Make clippy mandatory (#99)
• Refactor cache params (#104)
• Use unwrap instead of ? in tests.
• Record.get_value() returning Result (#115)
• Moved files (#122)
• Inject pipeline_details (#123)
• Make minor improvements to CLI experience. (#135)
• Remove debug messages (#141)
• Make run the default command (#142)
• Refactor indexes into multiple dbs (#167)
• Use 1.64.0 (#177)
• Introduce array types, uint and remove record type (#175)
• Refactor ingestion to connectors & ingestor (#179)
• Introduce a few orchestrator improvements (#184)
• Switch to stable (#188)
• Remove duplicate get_key method (#195)
• Remove an unnecessary unwrap (#196)
• Fix channel & safe generation (#198)
• Bulid secondary index key only from values. (#197)
• Use ipv4 address as default for grpc (#203)
• Update nightly benches script (#201)
• Re-export get_connector (#208)
• Reorganize files (#261)
• Add license (#262)
• Remove secondary_indexes from Schema (#263)
• Fix snowflake test (#303)
• Add openssl install instruction to README (#312)
• Delete secondary indexes on record deletion (#315)
• Fetch query results chunk by chunk when calculating intersection (#321)
• Get rid of prost-wkt (#333)
• Add timeout limit for github actions (#329)
• Serialize order_by to an object (#360)
• Add gRPC and REST load testing script (#355)
• Remove unnecessary PortHandle argument from sink set_input_schema() (#373)
• Use same Field serialization across modules (#363)
• Implement tests in dozer admin (#328)
• Restructure dozer config yaml (#352)
• Use custom composite key serialization instead of bincode (#381)
• Modify CI tasks on main (#402)
• Check download script exit status (#408)
• Remove values field of Schema (#409)
• Fix new clippy lints in rust 1.66 (#411)
• Remove FieldType::Null (#410)
• Remove spurious sort options test case (#419)
• Remove SortDirection from SortedInverted index definition (#421)
• Add test for CommonService (#426)
• Add cache e2e tests (#418)
• Run test that depends on mongodb behind a feature flag (#428)
• Remove dozer-admin config in root folder (#430)
• Use mongodb as optional dependency instead of dev dependency (#432)
• Move EventType to dozer.types (#444)
• Remove Null from field type proto (#443)
• Remove unused import (#442)
• Propagate error to process exit status (#448)
• Make executor core traits Debug (#480)
• Remove dag executor termination barrier (#481)
• Improve configuration experience (#472)
• Template update and fix readme (#457)
• Check home_dir in init and clean (#489)
• Update sample to use init command (#491)
• Refactor RecordWriter to support multiple strategies (#505)
• Disable e2e tests (#512)
• Bundle script setup dozer-admin + ui (#485)
• Disable e2e tests (#520)
• Remove unused fields in StateWriter (#519)
• Handle internal server error gracefully (#525)
• Refactor scalar functions error reporting and validation + LENGTH() / CONCAT() / TRIM() implementation (#523)
• Test both single process and separate process use scenario of dozer (#526)
• Implement separate to_[type]() for Field (#531)
• Add Field::as_* and Field::to_* unit tests (#533)
• Publish docker images and upload to s3 (#528)
• Don't build dozer in test execution. Use env var for finding binary. (#544)
• Include general workflow for discord (#559)
• Noify discord (#563)
• Check for duplicate connection name in AppSourceManager (#564)
• Change some unwraps to expects to explain why it's safe (#574)
• Ignore dozer-tests/* when counting coverage (#575)
• Enable building pre-release on main (#577)
• Add description for docs generated (#594)
• Remove some Path to str conversion (#596)
• Remove some unnecessary clones (#605)
• Delete unused source file (#606)
• Rename dozer init to dozer migrate (#595)
• Add authenticate_server_reflection flag (#597)
• Improve progress bar experience and error messaging (#615)
• Fix primary key logic (#618)
• Fixed v1 to original link (#629)
• Update proto file as matching update from dozer-types (#631)
• Update inline documentation (#645)
• Check name only contract is to be decoded (#652)
• Tag preleases as dev and releases as latest (#659)
• Add pipeline and container for building and running dozer-tests on Buildkite (#643)
• Include tag in Discord release notification (#660)
• Enable coverage on PR (#663)
• Download prebuilt grcov from public S3 bucket and enable cache (#666)
• Bump sqlparser to v0.30.0 (#686)
• Fix typo query_sqllite to query_sqlite (#693)
• Support for generic schema context in Sink, Processor and Source factories (#705)
• Expand sql tests (#706)
• Replace field_extract_* macros with Field.to_* in aggregators (#732)
• Delete useless code in planner (#735)
• Fix formatting issues (#743)
• Add release script copy for snowflake (#747)
• Don't cancel jobs on main to avoid red crosses (#777)
• Prepare release v0.1.6 (#796)
• Restore examples (#797)
• Remove duplicate schema and context in DagSchemas (#804)
• Panic hooks prints backtrace when it's enabled (#819)
• Fix dag issues (#818)
• Update docs (#822)
• Update dozer-tests/README.md (#827)
• Update with reformatted config (#823)
• Integrating sql planner (#811)
• Add debug assertion for empty primary index (#820)
• Include github token when installing protoc on ci (#835)
• Improve Join processor errors (#831)
• Include health in reflection (#828)
• Move SourceStates to dozer-types (#871)
• Remove unused connector function (#852)
• Make lmdb cache query module private (#878)
• Change diagram (#885)
• Remove redundant para (#886)
• Remove unused struct OperationEvent (#832)
• Add github star line (#892)
• Don't hold AccessFilter in CacheReader. Don't hold CacheReader or ApiEndpoint in ApiHelper (#894)
• Fix get of table info (#901)
• Remove unused dependency that caused a cargo warning (#915)
• Refactor orchestrator and implement new connectors (#932)
• Add Source::can_start_from with placeholder implementation (#929)
• Use pull_request_target instead (#977)
• Remove review comment (#978)
• Introduce python feature flag (#982)
• Remove python udf e2e test (#987)
• Update connectors readme table layout (#943)
• Fix proto definitions (#991)
• Update discord notification format of pull requests. Also notify issue comments. (#1003)
• Include source error in ListingPathParsingError (#1005)
• Print sink processing speed, in number of operations per sec (#1006)
• Remove cargo test connector_e2e_ because there's no test. (#1007)
• Don't fail discord notification when PR has no comment (#1026)
• Refactor grpc types (#1036)
• Timestamp difference (#1096)
• Add authors to packages (#1102)
• Move SnapshottingDone out of Operation so processors don't have to know it.(#1103)
• Don't create default eth provider (#1108)
• Remove an unused Arc<RwLock> (#1106)
• Provide context to ApiErrors (#1109)
• Add installation to and remove docker instructions from README (#1110)
• Add remove unnecessary libraries remove (#1113)
• Don't run coverage on pull_request_target (#1116)
• Use 16 cores runner for coverage and release, 4 cores runner for lint (#1118)
• Use correct proto include path to avoid rebuilding every time (#1115)
• Fix docker image publishing steps in release workflow (#1120)
• Remove some unnecessary clones (#1131)
• Fix clippy (#1138)
• Add missing ignore flags (#1144)
• Remove TableInfo::table_name and TableInfo::id. Remove parameter of Connector::get_tables. (#1145)
• Include security (#650)
• Bump sqlparser v0.31.0 (#1137)
• Use LmdbMap and LmdbMultimap instead of raw database in cache (#1156)
• Remove checkpoint from Sink (#1171)
• Remove Record::get_hashed_primary_key and use primary key instead (#1191)
• Bring back debug schema consistency check. Remove complex type used in test (#1187)
• Introduce another e2e test (#1194)
• Update readme (#1222)
• Remove unnecessary debug statements (#1232)
• Inlcude context in ListingPathParsingError (#1186)
• Include links to samples (#1241)
• Rename TableInfo to ListOrFilterColumns and move to connector module (#1246)
• Include links to samples (#1241) (#1214)
• Use full commit sha as docker image tag (#1238)
• Add specification for connectors (#1251)
• Rename admin types (#1276)
• Hint cache_max_map_size config in error message (#1275)
• Handle errors properly (#1279)
• Log datafusion read error (#1293)
• Remove unused id in PostgresConnector (#1298)
• Refactor SQL operations unit tests with proptest (#1287)
• Print field name on schema mismatch (#1300)
• Check map size before creating lmdb environment (#1303)
• Ignore sql join sample integration test because of lmdb crash (#1308)
• Remove dozer-storage dependency from dozer-core (#1319)
• Enable sql join e2e test (#1339)
• Add deb name to orchestrator cargo.toml (#1338)
• Remove unused spin loop in run_all. Sleep in spin loop of api thread. (#1341)
• Round cache_max_map_size to multiple of page size (#1347)
• Change an error log to debug log (#1343)
• Include env variables in cache benchmark (#1350)
• Fix duplicate sources (#1353)
• Run integration test in Github instead of buildkite (#1372)
• Remove protobuf dependency from deb (#1383)
• Don't cancel e2e runs on main (#1391)
• Use CacheWriteOptions instead of ConflictResolution (#1393)
• Use a single tokio runtime for everything (#1417)
• Use correct runtime for internal pipeline thread (#1418)
• Refactor as dozer-log crate (#1420)
• Add deb generation to release process (#1432)
• Remove unnecessary workspace.exlude in Cargo.toml (#1430)
• Make AccessFilter::fields optional (#1436)
• Only build dozer binary in release workflow (#1447)
• Remove dozer-admin package (#1448)
• Reexport two actix crates from dozer-api (#1451)
• Add get status method to admin proto service (#1450)
• Change error message for CustomTypeNotSupported (#1459)
• Remove DCO (#1470)
• Add dozer-log-python README and CI (#1471)
• Move log related code to dozer-types and dozer-log. (#1474)
• Fix dozer-log-python cross compilation (#1475)
• Don't run dozer-log-python workflow on PR and pushes to main (#1478)
• Rename admin to cloud (#1477)
• Remove outdated example (#1481)
• Add CI for publishing doer-log-js (#1476)
• Rename dozer_log to pydozer_log (#1482)
• Remove unused dependency fp_rust (#1486)
• Mention Dozer Log bindings in README (#1487)
• Rename pyproject name from dozer-log-python to pydozer_log (#1485)
• Change cloud command and make minor changes in cloud proto (#1489)
• Remove FieldBorrow (#1501)
• Let actix web handle json deserialization (#1504)
• Put the table prints behind "info" (#1509)
• Add cloud feature flag (#1513)
• Recover update_record_when_primary_changes test (#1517)
• Upgrade deltalake to 0.8.0 (#1531)
• Create documentation_improvement.md issue template (#1542)
• Don't write ansi color to file (#1545)
• Add a missing cloud feature gate (#1552)
• Add HomeDir::list_endpoints (#1554)
• Remove status update method from internal pipeline server (#1560)
• Json arrow conversion support (#1514)
• Update proto model matching dozer types (#1563)
• Call start_dozer on dozer cloud update (#1556)
• Add grpc endpoint for cloud to support develop ui (#1565)
• Remove unused optional in cloud.proto (#1576)
• Add delete files (#1577)
• Rename "revision" to "deployment" (#1579)
• Don't show progress bar or log color when not outputting to tty (#1580)
• Use dev environment endpoint as the default cloud target url (#1581)
• Remove logging to file (#1589)
• Rename dozer-orchestrator to dozer-cli (#1592)
• Add repository url to cloud proto and rename id to app_id (#1594)
• Include "x-dozer-server-name" in response header (#1599)
• Update Company Response metadata (#1603)
• Show status of all api servers (#1601)
• Add token param incase access config endpoint by Rest URL at server 8082 (#1615)
• Emit special error message for MapFull (#1618)
• Remove maturin sccache because it keeps failing (#1623)
• Cloud api endpoint now includes protocol (#1614)
• Remove extra http:// before cloud deployment endpoint (#1625)
• Add instance type to cloud config (#1629)
• Rollback changes for kafka connector feature flag (#1639)
• Read metadata during cache dump transaction creation (#1641)
• Add kafka to docker image (#1644)
• Enhance response for UI render (#1648)
• Add name fn for ProcessorFactory for UIDag in Cloud (#1658)
• Update default cloud target url (#1667)
• Send CI env to metadata (#1669)
• Add error message for wrong !Table config format for Parquet and CSV (#1675)
• Run CI scripts on merge groups instead of on pushes to main (#1683)
• Group CUD operation in cache/source as single metric and differentiate by label instead of 6 separate metric (#1684)
• Add id in processor factory to support query metric on UI (#1685)
• Fix typo in source manager app test (#1700)
• Use Utf8Path in HomeDir (#1702)
• Login to aws when running coverage job (#1708)
• Add more context for metric api-latency (#1713)
• Print in the schema if a field is primary key (#1717)
• Set bucket for histogram (#1723)
• Rename num replicas to num api instances (#1725)
• Edit kafka feature with snowflake docker image release flow (#1730)
• Remove outdated file (#1741)
• Update commands descriptions (#1739)
• Synchronize schema change (#1744)
• Added toolchain toml to pin version (#1749)
• Enhance label metrics for api_count and api_requests (#1752)
• Rename more migrate to build (#1755)
• Remove deprecated command line arguments (#1757)
• Bump semver from 6.3.0 to 6.3.1 in /dozer-log-js (#1735)
• Remove Schema::identifier (#1776)
• Update geo to use latest stable Rust (#1785)
• Remove unused trait ExpressionExecutor. (#1787)
• Include column name in InvalidColumn error (#1791)
• Update arrow version and its ecosystem to 42 (#1793)
• Remove unused function ProcessorRecord::extend_referenced_fields (#1788)
• Change make_from! in from_arrow to func to improve readability (#1792)
• Remove unused Orchstrator trait (#1797)
• Split ApiInitError out from ApiError (#1798)
• Remove AppSourceId which is no longer used (#1803)
• LogClient automatically retries on network error (#1811)
• Add SourceFactory::get_output_port_name to simplify ui graph generation (#1812)
• Add phase and last_error to DeploymentStatus (#1818)
• Start 0.1.33 dev (#1828)
• Add autdit field created_at updated_at (#1836)
• Rename err_threshold to error_threshold. Remove unused parameter. (#1837)
• Add "persist" message to pipeline (#1841)
• Move Epoch from dozer-types to dozer-core (#1843)
• Start v0.1.34 dev (#1845)
• Remove dead code (#1853)
• Start v0.1.35 dev (#1870)
• Remove short form of enable_progress because it's conflicting with dozer cloud (#1876)
• Enable push_events by default (#1877)
• Add Access deserialization test for All variant (#1878)
• Add flag to disable live ui for local development (#1880)

Performance

• Don't use bincode comparator for single field sorted inverted index (#233)
• Remove unnecessary clones in gRPC service (#353)
• Only consult epoch manager when commit is needed (#508)
• Remove RwCache::insert_schema so we don't read the db on every insert (#947)
• Remove usage of nested lmdb transaction (#1084)
• Optimize cache insertion and query when schema is append only (#1192)
• Put indexing thread pool to idle if indexes are up to date (#1320)
• Use MDB_APPEND when possible (#1362)
• Replace Vec<u8> with u64 in join lookup table to save memory (#1795)
• Optimize memory usage for Lifetime (#1796)

Refactor

• Refactor e2e (#64)
• Improve cache interface (#83)
• Build Processing logic on Schema update (#92)
• Dynamic schema support for groupby (#95)
• Remove references to lmdb::Database (#100)
• Use dynamic query value instead of field (#117)
• Migrated to thiserror for dozer-sql and dozer-core (#124)
• Introduce typed errors (#125)
• Refactor errors and reuse exported libs (#126)
• Encapsulate arc in lmdb-sys and tests cleanup (#134)
• Change IndexDefinition from struct to enum (#147)
• Remove state store (#180)
• Unify the Expression builders (#160)
• Change reader and writer intialization. (#212)
• Remove rocksdb and global counter (#235)
• SUM, COUNT Aggregator like scalar function (#245)
• Use a single tokio runtime for apis (#260)
• Implement explicit type conversion for gRPC methods. (#302)
• Sum aggregator (#300)
• Refactor features usage in tests (#306)
• MIN/MAX aggregator (#346)
• Aggregator tests (#376)
• Aggregation execution (#384)
• Use static schema (#385)
• CacheSink respects commit (#407)
• README + Examples (#345)
• Minor fix with folder structure (#427)
• Remove lmdb abstraction in dozer-core (#456)
• Use Pipeline builder (#469)
• Move dag executor implementation to separate files (#490)
• Remove out-of-context arguments in Sink::prepare() (#492)
• Added error msg for primary key missing/mismatching (#513)
• Readme with banners + remove unused readmes (#610)
• Discord notification (#694)
• Put pipeline_map into QueryContext (#731)
• Clean untagged ECR Images (#727)
• Expression parsing refactor to support nested aggregations (#715)
• Only persist checkpoint states of relevant sources. (#774)
• Use daggy for the underlying data structure of Dag (#790)
• Simplify Dag interface and move NodeHandle inside NodeType (#798)
• Don't try to get schema of unconnected ports (#801)
• Make LmdbRoCache and LmdbRwCache Send and Sync (#821)
• Split execution dag construction and execution (#842)
• Move metadata to DAG node. Don't create new node storage until execution (#846)
• Remove unused struct PipelineDetails (#883)
• Rename data fusion connector to object store (#881)
• Parse generated proto file on TypedService construction (#895)
• Refactor admin models (#888)
• Cleanup running flag and excessive Ingestor cloning (#916)
• Reduce duplication of code & more clone prevention (#922)
• Make tables vector required for connector start (#1058)
• Simplify schemas fetch query (#1114)
• Pipeline memory storage (#1135)
• Comparison errors (#1163)
• Drop multi schema support (was not correct anyway) in cache (#1159)
• Finalize Connector interface (#1233)
• Average (#1262)
• Product Processor parallel join execution (#1180)
• Stop using lmdb NO_LOCK flag. (#1337)
• Remove Record::version. Add version to CacheRecord. (#1392)
• Asyncify connectors (#1409)
• Store schema of sinks separately (#1423)
• Don't check for insert conflict when schema has no primary index (#1452)
• Create an api directory for every endpoint (#1497)
• Start a new migration when dozer restarts (#1503)
• Arrow conversion function (#1532)
• Remove raw config from cloud deploy API payload (#1533)
• Change errors structure for cloud (#1568)
• Add proper error handling for cloud service grpc errors (#1569)
• Traits in dozer-core return BoxedError instead of ExecutionError (#1595)
• Improve CLI experience (#1703)
• Show more information when configuration not found (#1734)
• Make config helper public (#1775)
• Remove nested record ref. Add ProcessorRecordStore (#1835)

Testing

• Fix snowflake test case (#258)
• Implement basic tests for postgres ingestion (#343)
• Disable snowflake tests (#506)
• Add basic e2e test framework (#522)
• Check gRPC health in e2e test (#527)
• E2e: Add endpoint and schema expectation (#529)
• E2e: Validate REST schema (#569)
• E2e: Rewrite framework to use prebuilt docker container (#567)
• Run all integration tests instead of only postgres (#782)
• Integrate docker containers to connectors unit tests (#980)
• Add tests for snowflake (#1016)
• Add tests for local storage (#1030)
• Run postgres tests on github actions and add snowflake support (#1037)
• Run e2e test locally on PR (#1239)
• Add ny taxi sample to e2e tests (#1263)
• LocalStorage basic test (#1290)
• Add DataReadyConnectorTest and implement LocalStorage (#1296)
• Add sql integration tests (#1282)
• Add postgres connector sample to e2e tests (#1278)
• Add postgres data type test (#1299)
• Add cud test suite and implement for postgres (#1309)
• Add integration test on Github runner (#1371)
• Implement integration test on Github runner (#1378)
• Add flights e2e scenario (#1425)
• Test dozer-log-python CI (#1473)
• Removed duplicate types (#1780)
• Provide a MySQL service for CI coverage tests (#1879)

Ci

• Introduce labeler workflow
• Introduce labeler workflow.v1

Deps

• Upgrade rustyline to 11.0.0 (#1613)

Dozer-admin

• Adapt new ui structure (#211)

Ref

• Groupby refactor + processor impl (#56)

Support

Contact us at https://getdozer.io