Skip to content

Releases: dingxin-tech/maxcompute-emulator

MaxCompute Emulator 1.1.0

Choose a tag to compare

@github-actions github-actions released this 17 Sep 08:42

ClickHouse protocol fixes and local reliability test controls.

  • Fix Tunnel Arrow ZSTD/LZ4 IPC compression and missing project/partition errors.
  • Add JSON download lifecycle logs with session hashes and request ranges.
  • Add 12 opt-in protocol fault effects with expiration, hit budgets and cancellation.
  • Support default/named quotas and QuotaNotExist / FlowExceeded tests.
  • Add optional ODPS v2/v4/STS signature, date and local read/write ACL verification.
  • Add public SQL TIMESTAMP_NTZ, MAP construction, nested/empty collections and numeric fixtures.
docker pull maxcompute/maxcompute-emulator:1.1.0
docker run --rm --platform linux/amd64 -p 127.0.0.1:8080:8080 \
  maxcompute/maxcompute-emulator:1.1.0 --listen 0.0.0.0:8080 --seed /opt/emulator/seed.sql

Use project test_project and point both service and Tunnel endpoints to the
mapped port. Default authentication remains permissive; fault injection is off.
See reliability controls for strict credentials, test-mode
network controls, quota configuration and /opt/emulator/types.sql.

Validation: Go race/vet in containers, Java SDK 0.61.2-public (33 tests,
including strict Storage reads and row-disconnect recovery), and a pinned public
C++ SDK Arrow wire probe (success, corrupt streams and one-shot recovery).
Publication runs these gates before pushing and reruns Java against the remote
image. Refer to the release Actions run for the published revision's results.

Limits: Full ClickHouse E2E has not been rerun for this release. The C++ probe
is a wire-contract test, not the complete SDK HTTP client. Production quota,
IAM/STS, routing and concurrency still require real MaxCompute acceptance.
The Java SDK's Long.MIN_VALUE rejection remains an explicit negative test.
Volume/Blob, Storage v1 and CDC remain unsupported. Session state is ephemeral;
Flink checkpoint recovery is not certified. Retain the original --project
when reusing a 1.0.0 DuckDB database; SQLite legacy data cannot be reused.

MaxCompute Emulator 1.0.0

Choose a tag to compare

@github-actions github-actions released this 14 Sep 08:35

The first community release of the Go + DuckDB rewrite.

  • SQL fixtures, table metadata, durable table IDs and STRING partition lifecycle.
  • Tunnel Protobuf/Arrow uploads and downloads, streaming upload, Upsert and instance results.
  • Storage API v2 Arrow reads/writes, session lifecycle and transactional batch commit.
  • CRC, bounded retry tracking, stale-session isolation and graceful shutdown.
  • Java SDK 0.61.2-public acceptance with Testcontainers.
docker pull maxcompute/maxcompute-emulator:1.0.0
docker run --rm --platform linux/amd64 -p 127.0.0.1:8080:8080 \
  maxcompute/maxcompute-emulator:1.0.0 --listen 0.0.0.0:8080 --seed /opt/emulator/seed.sql

Images are Linux amd64. Configure ODPS and Tunnel endpoints to the mapped port, project test_project, schema default, and dummy credentials.

Migration: main is now Go; the previous Spring Boot/SQLite implementation is preserved on legacy. Existing old image tags remain available. SQLite databases are incompatible; recreate SQL fixtures. Transfer sessions and retry history are not persisted across restarts.

Scope: local/CI testing; authentication signatures and permissions are not validated. Storage v1, Volume/Blob, CDC, explicit Schema management and column schema evolution are not supported. Flink standalone bounded uploads were verified; the tested connector's coordinator mode can finish without committing, and checkpoint recovery is not certified. Full ClickHouse engine integration is not claimed.

See README and docs/release-readiness.md for the complete compatibility matrix.

MaxCompute Emulator 0.0.7

Choose a tag to compare

@dingxin-tech dingxin-tech released this 07 Jan 12:33
v0.0.7

update version to 0.0.7

MaxCompute Emulator 0.0.6

Choose a tag to compare

@dingxin-tech dingxin-tech released this 14 Aug 02:12
d752632
Merge pull request #1 from dingxin-tech/tpch_version

use tpch version as default version

MaxCompute Emulator 0.0.5-tpch

Choose a tag to compare

@dingxin-tech dingxin-tech released this 26 Jul 02:41

CHANGELOG

v0.0.5-tpch

Based on the v0.0.5 version, this version has built-in TPCH-mini data set. This dataset was sampled from the full TPCH dataset with a sampling ratio of 0.01.

v0.0.5-tpch

该版本在 v0.0.5 版本的基础上,内置了 TPCH-mini 数据集。这个数据集是从完整 TPCH 数据集取样的,取样比例为 0.01。

MaxCompute Emulator 0.0.5

Choose a tag to compare

@dingxin-tech dingxin-tech released this 26 Jul 02:39

CHANGELOG

v0.0.5

new function

  • Added support for MaxStorage to read MaxCompute partition table (preview version, this function has not been fully verified)
  • The tunnel upsert function now supports any specified compress option (originally, only raw can be specified)

v0.0.5

新功能

  • 新增支持 MaxStorage 读取 MaxCompute 分区表(预览版,该功能未经过完整验证)
  • 现在 tunnel upsert 功能支持任意指定 compress option 了(原只能指定 raw)

MaxCompute Emulator 0.0.4

Choose a tag to compare

@dingxin-tech dingxin-tech released this 05 Jun 07:43

This is a patch version addressing several issues encountered while reading non-partitioned MaxCompute tables using MaxStorage (Storage API) (partitioned tables are currently unsupported). We have fixed these issues and ensured that the emulator can support the current version of the Presto-Connector for end-to-end testing.

New Features

  • Added support for different table splitting types in MaxStorage (currently only single split is supported, regardless of the table splitting method).

Fixes

  • Fixed an error that occurred when creating tables without primary keys.
  • Fixed an issue where table names and column names were not being converted to uppercase upon creation, to maintain the emulator's case-insensitivity feature. (Ideally, all case conversions should be removed).
  • Fixed an error that occurred when reading empty tables in MaxStorage.
  • Fixed an error that occurred when reading BIGINT type data in MaxStorage.

这是一个修复版本,我们在使用 MaxStorage(Storage API)读取 MaxCompute 非分区表时遇到了一些问题(分区表目前还不支持)。
我们修复了这些问题,并确保了模拟能够支持目前版本的 Presto-Connector
能够进行E2E测试。

新功能

  • 新增对 MaxStorage 不同切表类型的支持(无论那种切表方式,目前仅支持单个Split)

修复

  • 修复在创建无主键表时报错的问题
  • 修复在建表时,没有将表名,列名转换为大写的问题,以维持Emulator不区分大小写的特性。(其实应该将所有的大小写转换去掉)
  • 修复 MaxStorage 在读取空表时报错的问题
  • 修复 MaxStorage 在读取 BIGINT 类型数据报错的问题

MaxCompute Emulator 0.0.3

Choose a tag to compare

@dingxin-tech dingxin-tech released this 30 May 09:53

Release Note for Version 0.0.3

New Features

  1. Upsert for Partition Tables:

    • Implemented upsert functionality for partitioned tables.
    • Partitions are marked during table creation and treated as primary keys to ensure partition uniqueness.
    • Note: For MaxCompute partitioned tables, the simulated behavior in SQLite will not actually create partitions.
  2. Preview of Storage API Read Interface:

    • Developed partial implementation of the Storage API read interfaces.
    • This is a preview version, and the complete and fully functional version will be released in the next update.

版本 0.0.3 更新说明

新功能

  1. 分区表的 Upsert 写入:

    • 实现了对分区表的 upsert(更新插入)操作功能。
    • 通过在建表时标记分区列,并将分区列视为主键的形式,实现了分区的唯一性。
    • 注意:对于 MaxCompute 中的分区表,在 SQLite 的模拟中不会实际创建分区。
  2. Storage API 读接口预览版:

    • 开发了部分 Storage API 读接口的内容。
    • 这是预览版,完整和功能齐全的版本将在下一个更新中发布。

通过这些新特性,版本 0.0.3大大增强了对分区表操作的支持,并为未来的Storage API读接口功能奠定了基础。我们期待在未来的版本中提供更多功能和改进。

MaxCompute Emulator 0.0.2

Choose a tag to compare

@dingxin-tech dingxin-tech released this 23 May 12:11

We have identified a critical issue in containerized environments where the dynamic nature of IP addresses and port mappings was causing tunnel-related endpoints to malfunction due to their default behavior of returning local addresses.

What's New in Version 0.0.2

To address this challenge, we are excited to introduce the /init endpoint in version 0.0.2. This new feature allows for the uploading of the container's current IP address and port to the server immediately after the container starts. This enhancement ensures that all tunnel-related interfaces function correctly within the dynamic landscape of containerized deployment.

How to Use

After deploying your container, make a post request to the /init endpoint with the current IP address and port details. This step integrates seamlessly into your deployment process, enabling smooth and reliable functionality of tunnel-related services.

We appreciate your feedback and are committed to continually enhancing our solutions. Thank you for your support.


我们发现了容器化环境中的一个关键问题,其中 IP 地址和端口映射的动态特性导致与Tunnel相关的端口由于返回本地地址的默认行为而发生故障。

0.0.2 版本的新增功能

为了应对这一挑战,我们很高兴在 0.0.2 版本中引入“/init”接口。这项新功能允许在容器启动后立即将容器当前的 IP 地址和端口上传到服务器。此增强功能可确保所有与Tunnel相关的接口在容器化部署的动态环境中正常运行。

如何使用

部署容器后,使用当前 IP 地址和端口详细信息向“/init”端点发出发布请求。此步骤无缝集成到您的部署过程中,从而实现隧道相关服务的平稳可靠的功能。

我们感谢您的反馈,并致力于不断改进我们的解决方案。感谢您的支持。

MaxCompute Emulator 0.0.1

Choose a tag to compare

@dingxin-tech dingxin-tech released this 23 May 05:07
ab90831

As of now, the project is in the Pre-Alpha Phase. Our inaugural release, version 0.0.1, primarily targets support for Flink integration testing. This release showcases the implementation of key interfaces, featuring:

SQL-related Interfaces
Table Metadata-related Interfaces
Tunnel upsert Interfaces


截至目前,该项目处于 Pre-Alpha 阶段。我们的首个版本 0.0.1 主要目标是支持 Flink 集成测试。此版本实现了一些主要接口,其中包括:

SQL相关接口
Table 元数据相关接口
Tunnel upsert 写入接口