[Feature] Introduce a new framework that supports scheduling for Formats and Processes #3924

LiangDai-Mars · 2025-11-11T07:11:08Z

Why are the changes needed?

To introduce a new framework that supports scheduling for a wider range of Formats and Processes

Brief change log

Introduce a new top-level service, ProcessService, responsible for managing the lifecycle of Processes.
Introduce a new plugin type ActionCoordinator to extend how TableProcess is created.
Introduce a new plugin type ExecutionEngine to extend how external Process is executed.

How was this patch tested?

Add some test cases that check the changes thoroughly including negative and positive cases if possible
Add screenshots for manual tests if appropriate
Run test locally before making a pull request

Documentation

Does this pull request introduce a new feature? (yes)
If yes, how is the feature documented? (not documented)

baiyangtx · 2025-11-20T07:27:41Z

amoro-ams/src/main/java/org/apache/amoro/server/process/ActionCoordinator.java

+
+  long getExecutorDelay();
+
+  TableProcess<TableProcessState> getTableProcess(TableRuntime tableRuntime);


The active process is managed by ProcessService or TableRuntime.

ActionCoordinator should be stateless.

baiyangtx · 2025-11-20T07:28:20Z

amoro-ams/src/main/java/org/apache/amoro/server/process/ActionCoordinatorScheduler.java

+  }
+
+  protected boolean hasAliveTableProcess(TableRuntime tableRuntime) {
+    TableProcess<TableProcessState> process = coordinator.getTableProcess(tableRuntime);


Should not fetch active process from coordinator.

baiyangtx · 2025-11-20T07:35:10Z

amoro-common/src/main/java/org/apache/amoro/process/TableProcessStore.java

+
+  interface TableProcessOperation {
+
+    TableProcessOperation insertTableProcess(TableProcessState state);


baiyangtx · 2025-11-20T07:35:37Z

amoro-common/src/main/java/org/apache/amoro/process/TableProcessStore.java

+
+    TableProcessOperation insertTableProcess(TableProcessState state);
+
+    TableProcessOperation updateTableProcess(TableProcessState state);


Refer to TableRuntimeStore.

Updatable fields.

Parameters

baiyangtx · 2025-11-20T07:46:15Z

amoro-common/src/main/java/org/apache/amoro/process/TableProcess.java


  protected final T state;
  protected final TableRuntime tableRuntime;
+  protected final TableProcessStore tableProcessStore;


Remove T state from TableProcess.

baiyangtx · 2025-11-20T07:53:59Z

amoro-ams/src/main/java/org/apache/amoro/server/process/executor/ExecuteEngine.java

+  String submitTableProcess(TableProcess<TableProcessState> tableProcess);
+
+  /** Cancel a table process in engine. */
+  ProcessStatus cancelTableProcess(


void tryCancel(String processIdentifier);

baiyangtx · 2025-11-20T07:59:27Z

amoro-ams/src/main/java/org/apache/amoro/server/process/ProcessService.java

+              && process.getStatus() == ProcessStatus.FAILED
+              && process.getState().getRetryNumber() < scheduler.PROCESS_MAX_RETRY_NUMBER
+              && process.getTableRuntime() != null) {
+            process.getState().addRetryNumber();


use TableProcessStore.

baiyangtx · 2025-11-24T13:27:14Z

amoro-ams/src/main/java/org/apache/amoro/server/process/ProcessService.java

+    private final Map<ServerTableIdentifier, TableProcess> activeTableProcess =
+        new ConcurrentHashMap<>();
+    private Map<String, String> properties;
+


Map<ProcessId, TableProcess>
Map<TableIdentifier, Set>.

baiyangtx · 2025-11-24T13:28:35Z

amoro-ams/src/main/java/org/apache/amoro/server/process/ActionCoordinatorScheduler.java

+  public void handleTableRemoved(TableRuntime tableRuntime) {
+    TableProcess process =
+        processService
+            .getTableProcessTracker()
+            .getTableProcessInstance(tableRuntime.getTableIdentifier());
+    if (process != null) {
+      cancel(tableRuntime, process);
+    }


ProcessService could do this.

baiyangtx · 2025-11-24T13:29:06Z

amoro-ams/src/main/java/org/apache/amoro/server/process/ActionCoordinatorScheduler.java

+          tableRuntime.getTableIdentifier(),
+          getAction());
+    } else {
+      TableProcess process = coordinator.recoverTableProcess(tableRuntime, processMeta);


processStore

baiyangtx · 2025-11-24T13:29:54Z

amoro-ams/src/main/java/org/apache/amoro/server/process/ActionCoordinatorScheduler.java

+  }
+
+  protected void recover(TableRuntime tableRuntime, TableProcessMeta processMeta) {
+    if (hasAliveTableProcess(tableRuntime)) {


All processMeta must be recovered.

baiyangtx · 2025-11-24T13:30:47Z

amoro-ams/src/main/java/org/apache/amoro/server/process/ActionCoordinatorScheduler.java

+    TableProcess existProcess =
+        processService
+            .getTableProcessTracker()
+            .getTableProcessInstance(tableRuntime.getTableIdentifier());
+    if (existProcess != null && existProcess.getId() == process.getId()) {
+      process = coordinator.retryTableProcess(process);
+      processService.retry(process);
+    } else {
+      LOG.warn(
+          "Detect no table process exists or exist table process id is different from retry process for table runtime: {}, skip retry {} action this time.",
+          tableRuntime.getTableIdentifier(),
+          getAction());
+    }


No need to check here.

baiyangtx · 2025-11-24T13:32:13Z

amoro-ams/src/main/java/org/apache/amoro/server/process/ProcessService.java

+  public void recover(TableRuntime tableRuntime, TableProcess process) {
+    synchronized (process) {
+      // TODO: init some status
+      tableProcessTracker.trackTableProcess(tableRuntime.getTableIdentifier(), process);
+      executeOrTraceProcess(process);
+    }
+  }


private method. same with below.

baiyangtx · 2025-11-24T13:38:44Z

amoro-ams/src/main/java/org/apache/amoro/server/process/executor/TableProcessExecutor.java

+      while (isTableProcessExecuting(status)) {
+        if (isTableProcessCanceling(tableProcess.getStatus())) {
+          LOG.info(
+              "Table process {} with identifier {} may have been in canceling, exit submit process.",
+              tableProcess.getId(),
+              externalProcessIdentifier);
+          return;
+        }
+        try {
+          Thread.sleep(DEFAULT_POLL_INTERVAL_MS);
+        } catch (InterruptedException e) {
+          throw e;
+        }
+        status = executeEngine.getStatus(externalProcessIdentifier);
+      }
+    } catch (Throwable t) {
+      if (t instanceof InterruptedException) {
+        LOG.info(
+            "Table process {} with identifier {} may have been interrupted by canceling process, exit submit process.",
+            tableProcess.getId(),
+            externalProcessIdentifier);
+        return;
+      }
+      status = ProcessStatus.FAILED;
+      message = t.getMessage();
+    }


The thread must set process status to a finnal status before it return.

baiyangtx · 2025-11-24T13:40:23Z

amoro-ams/src/test/java/org/apache/amoro/server/process/TestExecuteEngine.java

+      System.out.println("Generated random number: " + randomNum);
+      System.out.println("Start loop: i from 0 to " + randomNum + ", sleep 5 seconds each time");
+      System.out.println("----------------------------------------");
+


Remove println, replace them with LOG.info

baiyangtx · 2025-11-24T13:42:24Z

amoro-ams/src/test/java/org/apache/amoro/server/process/TestExecuteEngine.java

+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+public class TestExecuteEngine implements ExecuteEngine {


Maybe MockExecuteEngine is better.

baiyangtx · 2025-11-24T13:43:09Z

amoro-ams/src/test/java/org/apache/amoro/server/process/MockActionCoordinator.java

+  public static final int PROCESS_MAX_POOL_SIZE = 1000;
+  private static final TableFormat[] DEFAULT_FORMATS = new TableFormat[] {TableFormat.PAIMON};
+
+  public static final Action DEFAULT_ACTION = new Action(DEFAULT_FORMATS, 0, "default_action");


MOCK_ACTION maybe better.

baiyangtx

I left some comments, ptal

baiyangtx · 2025-12-03T08:01:44Z

amoro-ams/src/main/java/org/apache/amoro/server/process/ActionCoordinator.java

+import org.apache.amoro.process.TableProcess;
+import org.apache.amoro.process.TableProcessStore;
+import org.apache.amoro.server.utils.SnowflakeIdGenerator;
+


class docs and method docs

baiyangtx · 2025-12-03T08:02:07Z

amoro-ams/src/main/java/org/apache/amoro/server/process/ActionCoordinatorScheduler.java

+import org.apache.amoro.server.table.TableService;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+


baiyangtx · 2025-12-03T08:05:54Z

amoro-ams/src/main/java/org/apache/amoro/server/process/ProcessService.java

+  }
+
+  public void register(TableRuntime tableRuntime, TableProcess process) {
+    synchronized (process) {


Synchronized by process maybe not work.

baiyangtx · 2025-12-03T08:09:26Z

amoro-ams/src/main/java/org/apache/amoro/server/process/executor/EngineType.java

+
+package org.apache.amoro.server.process.executor;
+
+public enum EngineType {


EngineType should be an extensible enumeration type, like Action

baiyangtx · 2025-12-03T08:10:35Z

amoro-ams/src/main/java/org/apache/amoro/server/process/executor/ExecuteOption.java

+
+package org.apache.amoro.server.process.executor;
+
+public enum ExecuteOption {


This class seems useless.

baiyangtx · 2025-12-03T08:12:11Z

amoro-ams/src/main/resources/mysql/upgrade.sql

+-- ADD external_process_identifier, retry_number, process_parameters to table_runtime
+ALTER TABLE `table_process`
+ADD COLUMN `external_process_identifier` varchar(256) DEFAULT NULL COMMENT 'Table optimizing external process identifier',
+ADD COLUMN `retry_number` int(11) NOT NULL COMMENT 'Retry times',


NOT NULL default 0

baiyangtx · 2025-12-03T08:16:06Z

amoro-common/src/main/java/org/apache/amoro/process/TableProcess.java

+    }
+  }
+
+  public boolean tryTransitState(


I think the transitState method should be in TableProcessStore.

baiyangtx · 2025-12-03T08:16:28Z

amoro-common/src/main/java/org/apache/amoro/process/TableProcessStore.java

+
+import java.util.Map;
+
+public interface TableProcessStore {


codecov-commenter · 2025-12-03T14:13:10Z

Codecov Report

❌ Patch coverage is 0% with 64 lines in your changes missing coverage. Please review.
✅ Project coverage is 22.09%. Comparing base (6166396) to head (8bf3551).
⚠️ Report is 20 commits behind head on master.

Files with missing lines	Patch %	Lines
...va/org/apache/amoro/process/TableProcessState.java	0.00%	33 Missing ⚠️
...in/java/org/apache/amoro/process/ProcessEvent.java	0.00%	13 Missing ⚠️
...in/java/org/apache/amoro/process/TableProcess.java	0.00%	11 Missing ⚠️
...in/java/org/apache/amoro/process/AmoroProcess.java	0.00%	5 Missing ⚠️
...n/java/org/apache/amoro/process/ProcessStatus.java	0.00%	2 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (6166396) and HEAD (8bf3551). Click for more details.

HEAD has 1 upload less than BASE

Flag BASE (6166396) HEAD (8bf3551)

core 1 0

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #3924      +/-   ##
============================================
- Coverage     28.84%   22.09%   -6.75%     
+ Complexity     3844     2461    -1383     
============================================
  Files           628      446     -182     
  Lines         50551    40952    -9599     
  Branches       6456     5767     -689     
============================================
- Hits          14580     9050    -5530     
+ Misses        34944    31144    -3800     
+ Partials       1027      758     -269

Flag	Coverage Δ
core	`?`
trino	`22.09% <0.00%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…rk test case

baiyangtx

LGTM

…ats and Processes (apache#3924) * process service poc * Refactor table process framework * Optimize and refactor the process service framework & support framework test case --------- Co-authored-by: dailiang <dailiang@bytedance.com> Co-authored-by: zhangyongxiang.alpha <zhangyongxiang.alpha@bytedance.com> Co-authored-by: baiyangtx <xiangnebula@163.com>

@turboFei

…lave mode is enabled. (#3846) * [Subtask]: Use a new configuration item to control whether master & slave mode is enabled. #3845 * [Subtask]: Use a new configuration item to control whether master & slave mode is enabled. #3845 * [AMORO-3850] Fix openapi sdk build and refine the package name (#3847) * rename the sdk pkg * version and artifactId * [HotFix] Change the uri configuration name in terminal (#3844) Change the uri configuration in terminal * [AMORO-3851] Separate spark-3.3 and spark-3.5 modules (#3848) * separate * java17 * build * GA * jdk17 tests * fix conflicts * <java.source.version>17</java.source.version> * JDK8 + Spark-3.5 * revert JDK17 * [AMORO-3857] Make the metadata file name conform to the Iceberg spec (#3858) Each version of table metadata is stored in a metadata folder under the table’s base location using a naming scheme that includes a version and UUID: <V>-<random-uuid>.metadata.json. * [AMORO-3850][FOLLOWUP] Add GA for OpenAPI SDK (#3870) [AMORO-3850][FOLLOWUP] Add GA for openAPI SDK * [AMORO-3864] Bump spark 3.5 version to 3.5.7 (#3860) Bump spark 3.5 version to 3.5.7 * [AMORO-3852][BUILD] Enhance build system (#3849) * Build project with fixed version check-latest: false GA * docs * shell * [AMORO-3851][FOLLOWUP] Using scala.binary.version for Spark dependency artifactId (#3861) Using scala.binary.version for dependency artifactId * [AMORO-3851][FOLLOWUP] Fix and simplify spark versions management (#3874) * Fix iceberg-spark artifactId in amoro-mixed-spark-3-common * fix ut conflicts * [AMORO-3863] Add script to reformat code (#3862) * format * spark profiles * [AMORO-3852] Using Maven Wrapper (#3879) * save * 3.9.9 * mvnw * slash * maven 3.9.11 * maven wrapper 3.3.4 * [AMORO-3880] Make amoro-openapi-sdk standalone (#3881) * [AMORO-3880] Make amoro-openapi-sdk standalone * save * [AMORO-3891] Bump netty version to 4.1.128.Final to fix CVE (#3892) Bump netty version to 4.1.128.Final to fix CVE * [AMORO-3890]Bump flink versions from 1.16.2/1.17.1/1.20.0 to 1.16.3/1.17.2/1.20.3 (#3889) bump * [AMORO-3880][FOLLOWUP] Fix pull request paths for OpenAPI SDK CI (#3903) * Bump orc-core version from 1.8.3 to 1.9.7 (#3900) * [AMORO-3883] Bump Paimon 1.1.1 to 1.2.0 (#3884) * Bump Paimon 1.1.1 to 1.2.0 * [AMORO-3883] Bump Paimon 1.1.1 to 1.2.0 --------- Co-authored-by: Xu Bai <xuba@apache.org> * [AMORO-3863][FOLLOWUP] Support to reformat pom (#3882) Support to reformat pom format trino * [AMORO-3872] Support to customize basic authentication implementation (#3871) * support to customize basic auth" * save * docs * common * address comments * nit --------- Co-authored-by: Xu Bai <xuba@apache.org> * [AMORO-3907] Bump kubernetes-client version to 6.13.5 (#3908) Bump kubernetes-client version to 6.13.5 * [AMORO-3875] Support to build on Spark Scala-2.13 (#3878) * scala binary version save bin profile GA revert paimon ams api compatiblity fix flink scala style save save rewrite by scala save conflicts dependency save save scala paimon ignore paimon idea * nit * nit * save * revert ci change * [AMORO-3875][FOLLOWUP] Fix optimizer-spark docker image build due to artifactId change (#3912) * fix save * test * Revert "test" This reverts commit 6c38dbd. * [AMORO-3863][FOLLOWUP] Enable format-mixed-format-trino profile on JDK17+ automatically (#3906) reformat * [AMORO-3848][FOLLOWUP] Prompt spark profile in docker build (#3904) Prompt spark profile in docker build * [AMORO-3918][INFRA] Add dependencies check CI (#3917) * [AMORO-3931] Exclude curator and zookeeper deps (#3932) * save deps exclude jline * nit * [AMORO-3945][DOCS] Add docs for REST API (#3944) docs combine * [AMORO-3873] Support Bearer/JWT authentication (#3905) * save * save * save * save * Save * asve * rename package * comments * basic * remove token from log * [AMORO-3938] change file_content_b64 column type to CLOB type (#3939) * [AMORO-3933] Fix Playground demo failure due `PartitionExpressionForMetastore class not found` (#3935) * Fix Playground demo failure due PartitionExpressionForMetastore class not found * Revert "Fix Playground demo failure due PartitionExpressionForMetastore class not found" This reverts commit fea6ddc. * add for runtime hive-exec --------- Co-authored-by: Xu Bai <xuba@apache.org> * Save the last completion time for each cleanup operation performed on each optimization table. (#3802) * Save the last completion time for each cleanup operation performed on each optimization table. # Conflicts: # amoro-ams/src/main/resources/mysql/upgrade.sql * Store the execution time of each cleanup operation for the optimization table in the table_runtime_state table. * fixup style --------- Co-authored-by: 张文领 <zhangwl9@chinatelecom.cn> * nit: remove duplicate code (#3957) NIT: Remove duplicate code * [AMORO-3873][FOLLOWUP] Rename rest auth bearer type to JWT (#3953) * [AMORO-3968] Update the thrift api compile command to use amoro shaded thrift (#3967) * relocate generated thrift code to use amoro shaded thrift * docs * save * profile * [AMORO-3961] Filter null key and value for Configurations::toMap (#3962) filter null key value * nit: remove unused code (#3959) remove unused code * [AMORO-3972][Core] Upgrade default Spark version from 3.3 to 3.5 (#3975) * [AMORO-3972][Core] Upgrade default Spark version from 3.3 to 3.5 This PR upgrades the default Spark version from 3.3 to 3.5 in the Amoro project. Changes: - pom.xml: Update spark.version from 3.3.4 to 3.5.7 and spark.major.version from 3.3 to 3.5 - pom.xml (hadoop2 profile): Update spark.version from 3.3.4 to 3.5.7 and spark.major.version from 3.3 to 3.5 - docker/build.sh: Update SPARK_VERSION from 3.3.3 to 3.5.7 - docker/optimizer-spark/Dockerfile: Update ARG SPARK_VERSION from 3.3.3 to 3.5.7 - .github/workflows/docker-images.yml: Update Spark optimizer matrix from 3.3.3 to 3.5.7 - README.md: Update Spark optimizer default version documentation Closes #3972 * Keep Spark 3.3 as default for hadoop2 profile Address review feedback from @turboFei: For hadoop2 profile, keep using the legacy Spark 3.3 by default for better compatibility with Hadoop 2.x. * Fix Spark version parameter name in README.md Address review feedback from @turboFei: Change -Dspark-optimizer.spark-version to -Dspark.version This was missed in PR #3874. * [AMORO-3966][Helm] Support custom volumes and volumeMounts (#3965) [Improvement][Helm] Support custom volumes and volumeMounts * [AMORO-3970] Update README.md with correct Spark versions for Mixed format (#3978) * [AMORO-3977] Combine amoro-site to prevent docs loose sync (#3979) * clone site * remove invalid soft links * docs soft link * Update site/README.md with clear structure documentation - Reorganized documentation to show the versioned and non-versioned content structure - Provided a clear directory tree representation of the site structure - Updated instructions for running the documentation site locally - Added section on testing both sites together 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Update site/README.md with clear structure documentation - Fixed references to make it clear this is part of the main repository - Updated section titles to match the new organization - Clarified paths for versioned and non-versioned content - Simplified local development instructions 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * github action * ignore * rat exclude site * test * test please * Revert "test please" This reverts commit 7c536fe. * label --------- Co-authored-by: Claude <noreply@anthropic.com> * [Improvement]: Disable verification for helm-unittest plugin installation (#3993) Disable verification for helm-unittest plugin installation * [AMORO_3990] Add package chart instructions to README (#3991) * [Feature] Introduce a new framework that supports scheduling for Formats and Processes (#3924) * process service poc * Refactor table process framework * Optimize and refactor the process service framework & support framework test case --------- Co-authored-by: dailiang <dailiang@bytedance.com> Co-authored-by: zhangyongxiang.alpha <zhangyongxiang.alpha@bytedance.com> Co-authored-by: baiyangtx <xiangnebula@163.com> * [AMORO-3981] Automatic generate amoro configuration docs (#3982) * auto generate ams config * add auto generate notice * junit 5 * convertToString * Update CONTRIBUTING.md --------- Co-authored-by: big face cat <731030576@qq.com> * [AMORO-3981][FOLLOWUP] Refine the duration to string format (#3987) * [AMORO-3981][FOLLOWUP] Refine duration type parse and convert to string * Revert "[AMORO-3981][FOLLOWUP] Refine duration type parse and convert to string" This reverts commit 877bcb9. * format duration with TimeUtils.formatWithHighestUnit * largest unit * [AMORO-1258] Support Zookeeper kerberos authentication (#3980) * [AMORO-1258] Support Zookeeper kerberos authentication * remove minkdc * Revert "remove minkdc" This reverts commit 466d632. * docs * [AMORO-3934] Manage com.fasterxml.jackson dependencies in dependencyManagement (#3936) * jackson version * deps * fasterxml.jackson.version --------- Co-authored-by: ZhouJinsong <zhoujinsong0505@163.com> * [AMORO-3866] Using shaded hadoop-client-api and hadoop-client-runtime for hadoop3 (#3983) hadoop client api and hadoop client runtime remove guava test replace exclude remove runtime * [AMORO-3632] Refine data expiration literal calculation for date type (#3964) * [Feature]: data-retention, add support for partition column type Date #3632 * [AMORO-3632]: data-retention, add support for partition column type Date * [AMORO-3632]: data-retention, add support for partition column type Date * [AMORO-3632]: data-retention, add support for partition column type Date * [AMORO-3632] Fix #3665: Prevent long overflow in Date expiration calculation and fix related tests --------- * [AMORO-3885] fix missing oss.endpoint for internal mixed_iceberg catalog (#3886) * fix missing oss.endpoint for internal mixed_iceberg catalog * fix missing oss.endpoint for internal mixed_iceberg catalog --------- Co-authored-by: ZhouJinsong <zhoujinsong0505@163.com> Co-authored-by: ConradJam <jam.gzczy@gmail.com> * [AMORO-2635] Enhance table partition files list performance (#4003) * [AMORO-2635] * UT * UT for fallback * address comments * TODO * [AMORO-3628] Add user logo wall on the home page. (#4010) * [AMORO-3628] Add users logo wall in the home page * update Co-authored-by: Claude * [AMORO-3804] Skip RUNTIME_CONTEXT_CACHE for TableMetaStore with local configuration (#4005) * [AMORO-3804] Skip RUNTIME_CONTEXT_CACHE for TableMetaStore with local configuration * return * test * configuration * create runtime context directly * remove unneeded tests * [AMORO-4011] Fix JUnit 4 tests skipped (#4017) * add back * test * revert test * deps * [AMORO-3775] Add support for metric-based refresh event trigger in TableRuntimeRefreshExecutor (#3776) [AMORO-3775] Add support for metadata-based refresh event in TableRuntimeRefreshExecutor * [AMORO-3998]support DATABASE HA SERVICE (#3997) support DATABASE HA SERVICE Co-authored-by: dailiang <dailiang@bytedance.com> * [Hotfix] Fix the deploy site GitHub workflow (#4020) * Fix deploy site GitHub workflow * Add publish information in the .asf.yaml * [Hotfix] Enable manual triggering for site deployment workflow (#4021) * [Hotfix]Refactor the build command in the `Publish Docker Image` GitHub workflow to reduce image size (#4023) Refactor the build command in the Publish DOcker Image GitHub workflow to reduce image size * [Hotfix] Fix typos (#4026) * [AMORO-3531] Drop support for java8 (#3899) drop support jdk8 Co-authored-by: Xu Bai <xuba@apache.org> Co-authored-by: ConradJam <jam.gzczy@gmail.com> Co-authored-by: ZhouJinsong <zhoujinsong0505@163.com> * [AMORO-3940] Flink config load will cause ClassCastException when dir… (#3941) [AMORO-3940] Flink config load will cause ClassCastException when directly return result of Yaml load Co-authored-by: ConradJam <jam.gzczy@gmail.com> * [AMORO-3943] Shade all third party classes for spark runtime (#3942) * relocate * filter * [AMORO-4027] Bump shade plugin to fix compilation failure on JDK 17 (#4028) (cherry picked from commit 83e5272) * [AMORO-3973] Support spark3.4 for mixed format (#4013) * Copy the code form spark3.5 * Optimize code * Maintain consistency in the references to scala-compiler and scala-library across Spark 3.5, Spark 3.4, and Spark 3.3 * fixup --------- Co-authored-by: 张文领 <zhangwl9@chinatelecom.cn> Co-authored-by: Xu Bai <tocreationbai@gmail.com> * [Hotfix] Try to fix site-deploy permission problem (#4025) * [AMORO-4022] [Improvement]: AMS Iceberg maintainer moved to the amoro-iceberg module (#4024) [AMORO-4022] AMS Iceberg maintainer moved to the amoro-iceberg module (#4022) * [AMORO-3994] Support Exposing AMS High Availability (HA) Status (#3996) [amoro-3994] Support Exposing AMS High Availability (HA) Status Co-authored-by: davedwwang <davedwwang@tencent.com> * [Feature]: Support LDAP Authentication for Dashboard Login (#4009) Co-authored-by: davedwwang <davedwwang@tencent.com> * [HotFix] Remove duplicate references to scala-library and scala-compiler (#4033) Remove duplicate references to scala-library and scala-compiler Co-authored-by: 张文领 <zhangwl9@chinatelecom.cn> * [HotFix] Shade third party classes for spark-3.4 runtime (#4032) Shade third party classes for spark runtime 3.4 Co-authored-by: 张文领 <zhangwl9@chinatelecom.cn> * [Subtask]: Use a new configuration item to control whether master & slave mode is enabled. #3845 * [Subtask]: Use a new configuration item to control whether master & slave mode is enabled. #3845 * [Subtask]: Use a new configuration item to control whether master & slave mode is enabled. #3845 * [Subtask]: Use a new configuration item to control whether master & slave mode is enabled. #3845 * [Subtask]: Use a new configuration item to control whether master & slave mode is enabled. #3845 * [AMORO-3998]support DATABASE HA SERVICE (#3997) support DATABASE HA SERVICE Co-authored-by: dailiang <dailiang@bytedance.com> * [AMORO-3994] Support Exposing AMS High Availability (HA) Status (#3996) [amoro-3994] Support Exposing AMS High Availability (HA) Status Co-authored-by: davedwwang <davedwwang@tencent.com> * [Subtask]: Use a new configuration item to control whether master & slave mode is enabled. #3845 * [Subtask]: Use a new configuration item to control whether master & slave mode is enabled. #3845 * [Subtask]: Optimize the description information for use-master-slave-mode. --------- Co-authored-by: wardli <wardli@tencent.com> Co-authored-by: Fei Wang <fwang12@ebay.com> Co-authored-by: ZhouJinsong <zhoujinsong0505@163.com> Co-authored-by: yeatsliao <liaoyt66066@gmail.com> Co-authored-by: Xu Bai <xuba@apache.org> Co-authored-by: xuzifu666 <xuzifu666@gmail.com> Co-authored-by: simonsssu <barley0806@gmail.com> Co-authored-by: zhangwl9 <1298877813@qq.com> Co-authored-by: 张文领 <zhangwl9@chinatelecom.cn> Co-authored-by: zhan7236 <76658920+zhan7236@users.noreply.github.com> Co-authored-by: Abhishek Pathania <theabhishekpathania@gmail.com> Co-authored-by: xykera <akrai9554@gmail.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: tcodehuber <tcodehuber@gmail.com> Co-authored-by: LiangDai-Mars <dd574852610@gmail.com> Co-authored-by: dailiang <dailiang@bytedance.com> Co-authored-by: zhangyongxiang.alpha <zhangyongxiang.alpha@bytedance.com> Co-authored-by: baiyangtx <xiangnebula@163.com> Co-authored-by: big face cat <731030576@qq.com> Co-authored-by: davedwwang <iverson89w@163.com> Co-authored-by: ConradJam <jam.gzczy@gmail.com> Co-authored-by: Xu Bai <tocreationbai@gmail.com> Co-authored-by: Jzjsnow <snow.jiangzj@gmail.com> Co-authored-by: leosanqing <liurongtong001@qq.com> Co-authored-by: leosanqing <stormleo@qq.com> Co-authored-by: Sebb <sebbASF@users.noreply.github.com> Co-authored-by: davedwwang <davedwwang@tencent.com>

dailiang and others added 2 commits November 7, 2025 11:19

process service poc

0824c33

Refactor table process framework

4c3630a

github-actions bot added module:ams-server Ams server module module:common labels Nov 11, 2025

LiangDai-Mars marked this pull request as ready for review November 11, 2025 07:11

LiangDai-Mars marked this pull request as draft November 11, 2025 07:11

LiangDai-Mars changed the title ~~Apache amoro process service poc~~ [WIP] Apache amoro process service poc Nov 11, 2025

LiangDai-Mars mentioned this pull request Nov 11, 2025

[WIP] process service poc #3913

Closed

baiyangtx marked this pull request as ready for review November 20, 2025 07:20

baiyangtx reviewed Nov 20, 2025

View reviewed changes

LiangDai-Mars force-pushed the apache-amoro-process-service-poc branch from 9258b2c to d75cbc1 Compare November 22, 2025 10:29

baiyangtx reviewed Nov 24, 2025

View reviewed changes

baiyangtx changed the title ~~[WIP] Apache amoro process service poc~~ [Feature] Introduce a new framework that supports scheduling for Formats and Processes Nov 25, 2025

LiangDai-Mars force-pushed the apache-amoro-process-service-poc branch 7 times, most recently from 0a67603 to 34cb61c Compare December 1, 2025 03:14

LiangDai-Mars force-pushed the apache-amoro-process-service-poc branch from 34cb61c to e23dd3c Compare December 3, 2025 07:54

baiyangtx reviewed Dec 3, 2025

View reviewed changes

LiangDai-Mars force-pushed the apache-amoro-process-service-poc branch 2 times, most recently from 9b9483d to 8bf3551 Compare December 3, 2025 13:58

Optimize and refactor the process service framework & support framewo…

fe102cb

…rk test case

LiangDai-Mars force-pushed the apache-amoro-process-service-poc branch from 8bf3551 to fe102cb Compare December 4, 2025 07:32

Merge branch 'master' into apache-amoro-process-service-poc

53cef63

baiyangtx approved these changes Dec 4, 2025

View reviewed changes

baiyangtx merged commit 1b03bb7 into apache:master Dec 4, 2025
8 checks passed


		long getExecutorDelay();

		TableProcess<TableProcessState> getTableProcess(TableRuntime tableRuntime);


		interface TableProcessOperation {

		TableProcessOperation insertTableProcess(TableProcessState state);


		TableProcessOperation insertTableProcess(TableProcessState state);

		TableProcessOperation updateTableProcess(TableProcessState state);


		package org.apache.amoro.server.process.executor;

		public enum EngineType {


		package org.apache.amoro.server.process.executor;

		public enum ExecuteOption {

[Feature] Introduce a new framework that supports scheduling for Formats and Processes #3924

[Feature] Introduce a new framework that supports scheduling for Formats and Processes #3924

Uh oh!

Conversation

LiangDai-Mars commented Nov 11, 2025 • edited by baiyangtx Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are the changes needed?

Brief change log

How was this patch tested?

Documentation

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

baiyangtx left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov-commenter commented Dec 3, 2025

Codecov Report

Uh oh!

baiyangtx left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

LiangDai-Mars commented Nov 11, 2025 •

edited by baiyangtx

Loading