1.13.0

This version has done a lot of refactoring in the streaming runtine, especially in source. The source node has been splitted into multiple nodes including connector, decompressor and decoder according to the source configurations like format, compressing and shared etc. This will help users to monitor rules in more detail, knowing which process either decode and decompress takes more time and other metrics, thus, we can be able to optimize in a finer grain. It will make it easier for developers to maitain and add more features into source without a need to maintain an all-in-one but complex source node. In this version, the source split only applies to MQTT source. We will expand the support to all source progressively in the following versions. And feedbacks are welcomed.

Streaming Runtime

• Dynamic Schema
  • feat: support merge schema by @Yisaer in #2502
  • feat: support merge schema for wildcard by @Yisaer in #2516
  • feat: support decode schemaless json by @Yisaer in #2500
  • feat: support runtime schema in rule explain by @Yisaer in #2545

• Refactor source to connector, decompressor and decoder and allow ordered concurrency. This should improve throughput a lot if the payload is big and decoding latency is long
• Refactor ping for connectors, provide more accurate info

SQL Syntax

• New function object_pick and kv/kvarray conversion functions
• feat: implement math functions by @carlclone in #2483
• feat: implement CONV function by @carlclone in #2506
• feat(func): add locale argument support by @carlclone in #2522

Extension

• JavaScript Function
  • feat(script): add javascript function by @ngjaying in #2549
  • feat(cli): add script support by @ngjaying in #2553
  • feat(script): add script import/export by @ngjaying in #2552

Configuration and Validation

• feat: support raw tls config by @Yisaer in #2550
• Introduce error code

All Changes

• feat: implement math functions by @carlclone in #2483
• fix: fix lookup plan column pruning by @Yisaer in #2494
• refactor(trial): catch sink errors by @ngjaying in #2499
• fix: fix sourceNode options by @Yisaer in #2501
• chore: add sniffio dependency for python plugins by @Yisaer in #2513
• feat: implement CONV function by @carlclone in #2506
• chore(deps): bump github.com/dvsekhvalnov/jose2g by @dependabot in #2514
• chore(deps): bump github.com/dvsekhvalnov/jose2go by @dependabot in #2515
• feat: support merge schema by @Yisaer in #2502
• feat: support merge schema for wildcard by @Yisaer in #2516
• docs(example): merge multiple stream by @ngjaying in #2518
• feat: support decode schemaless json by @Yisaer in #2500
• fix(decode): perf update by @ngjaying in #2519
• chore: update httppush.json file by @oucb in #2520
• fix(rest): return 404 when rule not found by @carlclone in #2505
• feat(func): add locale argument support by @carlclone in #2522
• fix: fix confKey API can't replace url/passwd by @Yisaer in #2538
• fix(trial): avoid rule id conflict by @ngjaying in #2539
• fix: fix sql reconnect by @Yisaer in #2541
• feat: support hide url password in log by @Yisaer in #2542
• fix: fix install plugin failed after upgrade by @Yisaer in #2543
• fix(doc): complete directory description by @carlclone in #2546
• feat: support runtime schema in rule explain by @Yisaer in #2545
• feat(script): add javascript function by @ngjaying in #2549
• refactor(component): extract import/export manager by @ngjaying in #2551
• feat(cli): add script support by @ngjaying in #2553
• feat(script): add script import/export by @ngjaying in #2552
• docs(script): add script support related doc by @ngjaying in #2556
• fix: fix memory leak if window condition unmatch by @Yisaer in #2559
• fix: fix schema explain by @Yisaer in #2555
• feat: support raw tls config by @Yisaer in #2550
• perf(sink): avoid reconstruct send param by @ngjaying in #2557
• feat: support object_pick function by @rextang in #2560
• fix: delete default username of redisPubSub by @retoool in #2566
• feat: support message processed metrics by @Yisaer in #2561
• feat: support connection native ping by @Yisaer in #2562
• test: add more test for lines format by @carlclone in #2570
• refactor: extract format reader from file source by @carlclone in #2571
• feat: support more kv functions by @rextang in #2575
• refactor: node runtime by @ngjaying in #2582
• fix: fix object function args is nil by @Yisaer in #2586
• fix: fix and test array function if has nil by @Yisaer in #2581
• feat: support errorcode framework by @Yisaer in #2577
• feat(node): split source connector and decoder by @ngjaying in #2583
• fix(functions): args validation of date_calc by @yai-dev in #2585
• fix: fix TestPing test by @Yisaer in #2595
• feat: support parser/plan/executor error code by @Yisaer in #2594
• feat: introduce undefined error code by @Yisaer in #2596
• feat: revise restart rule by @Yisaer in #2597
• feat: support neuron/kafka ping by @Yisaer in #2591
• feat(node): ordered concurrency support by @ngjaying in #2592
• fix(topo): avoid plan twice by @ngjaying in #2600
• fix(stats): fix get metrics wrong pos by @ngjaying in #2601
• fix: fix source kafka plugin ci build by @Yisaer in #2604
• fix: fix schemaless decode error by @Yisaer in #2605
• feat: add error code by @Yisaer in #2598
• refactor(node): split decompressor op by @ngjaying in #2602
• fix(topo): subtopo runtime fix by @ngjaying in #2609
• feat: add stream/table/rule/confkey error code by @Yisaer in #2607
• feat: support expr validate by @Yisaer in #2608
• fix: fix protobuf oneof decode by @Yisaer in #2610
• fix(docs): correct typo by @xiaozexuan in #2612
• feat(node): split shared connection as a node by @ngjaying in #2613
• fix: fix rest and sql error and log by @Yisaer in #2620
• docs: add error code doc by @Yisaer in #2621
• fix: skip check cert if not necessary by @Yisaer in #2623
• fix(node): avoid excessive log when enable cp by @ngjaying in #2622
• feat: revise mqtt tls error msg by @Yisaer in #2626
• fix: fix gob encoding by @Yisaer in #2639
• fix(func): fix ln function by @ngjaying in #2630
• fix: fix status message by @Yisaer in #2636
• feat: let stmt validate by checker by @Yisaer in #2638
• feat: support validate conf before creating by @Yisaer in #2635
• feat: introduce failpoint and fix resend error by @Yisaer in #2642
• feat: check rule before creating it by @Yisaer in #2624
• fix: fix sql error status by @Yisaer in #2643
• fix: check redis connector db by @Yisaer in #2644
• feat: support validate rule id by @Yisaer in #2645
• fix: support validate rest sink method by @Yisaer in #2647
• fix: add pynng version by @Yisaer in #2648
• fix: lock pynng version 0.7.3 by @Yisaer in #2649
• feat(service): desc func return addr by @ngjaying in #2651
• fix: lock pynng into 0.7.2 by @Yisaer in #2654
• ci(build): add workflow_dispatch to build packages by @Rory-Z in #2655
• fix(docs): clean up en content by @ngjaying in #2656
• fix: fix sink sql for oracle by @Yisaer in #2657
• fix(analyzer): correct embedded analytic funcs by @ngjaying in #2653
• fix: fix zh doc by @Yisaer in #2658
• fix: fix buffer length metrics by @Yisaer in #2659
• fix(service): delete schemaless func by @ngjaying in #2660
• fix(build): remove script from default by @ngjaying in #2662
• fix: fix converter schema display by @Yisaer in #2663
• fix: fix conv validate by @Yisaer in #2664
• fix: fix cert options by @Yisaer in #2671
• fix(topo): avoid wrong topo open state by @ngjaying in #2672
• fix(topo): catch recover rule error by @ngjaying in #2673
• fix: fix influx sink configure by @Yisaer in #2670
• fix(rest): sink should return unrecoverable error by @ngjaying in #2676
• docs: fix Chinese doc errors for substring func by @Dream95 in #2661
• fix: fix wildcard flag and default fastjson decode by @Yisaer in #2677
• feat: union mqtt sink tls error behavior by @Yisaer in #2675
• ci(docker): add full tag by @ngjaying in #2682
• fix: tdengine fields setting problem by @gooker in #2681
• refactor: split shared source schema support by @ngjaying in #2678
• docs: revise sqlserver doc by @Yisaer in #2683
• fix: fix neuron ping by @Yisaer in #2686
• chore: move the plugin yaml by @ngjaying in #2684
• fix: fix kafka sasl ping by @Yisaer in #2687
• fix: fix explain unexist rule by @Yisaer in #2685
• fix(func): last_value for nil array by @ngjaying in #2690
• fix: fix ruleID check by @Yisaer in #2693
• chore(deps): bump github.com/jackc/pgx/v4 from 4.18.1 to 4.18.2 by @dependabot in #2699
• chore(deps): bump github.com/jackc/pgx/v4 from 4.18.1 to 4.18.2 in /extensions by @dependabot in #2698
• fix: fix join exit check by @Yisaer in #2695
• feat: add log when remove metrics by @Yisaer in #2701
• fix: remove cpu consuming log by @Yisaer in #2697
• chore(deps): bump google.golang.org/protobuf from 1.28.0 to 1.33.0 in /internal/converter/protobuf/test by @dependabot in #2705
• chore: update sql.json file by @oucb in #2703
• fix(config): should not export empty config by @ngjaying in #2702
• docs(func): remind of mem usage for lpad/rpad by @ngjaying in #2708
• fix: fix rule id check by @Yisaer in #2709
• fix: revise ping message by @Yisaer in #2710
• fix: remove neuron ping by @Yisaer in #2711
• fix: revise mqtt source on msg behavior by @Yisaer in #2712

New Contributors

• @rextang made their first contribution in #2560
• @xiaozexuan made their first contribution in #2612
• @Dream95 made their first contribution in #2661
• @gooker made their first contribution in #2681

Full Changelog: 1.12.1...1.13.0