Any plan to implement a job scheduler?

[zhicwu]

Is there any plan to implement a (distributed) job scheduler like in MS SQL server? It's about the ability to define job and schedule, and attach schedule to specific job(s).

The use case is to use ClickHouse as a data platform not only for managing data but also scheduling jobs for ELT and DB maintenance etc., for example:

-- use database/schema as namespace for jobs and schedules
use mydb;

-- show user objects
show schedules;
show jobs;

-- create schedules
drop schedule if exists `nightly`;
create or replace schedule `nightly` as 'H 23 * * 1-5';
drop schedule if exists `5 minutes later`;
create or replace schedule `5 minutes later` as '* * * * *' option(times=1, start_time=now() + interval 5 minute);

-- create jobs with schedule attached
drop job if exists `full load`;
-- create job `an external job with supported file extension` as script '/tmp/my-script.sh';
-- create job `a java job` as java (...);
create or replace job `full load` as sql (
  execute `backup target table`; -- chaining
  truncate table target_table;
  insert into target_table select current_query_id(), * from url(...)
) options(retry=0, transaction=new, schedule='nightly');

[tbragin]

As far as I know there is no effort to implement a generic task scheduler in ClickHouse as you are describing (see 2024 roadmap). Of course, things could change and community PR could show up with this functionality if someone happened to be working on it, but I don't see anything there currently.

The closest functionality I can think of based on your use case is refreshable that may be helpful is "refreshable materialized views", which allows refreshing MVs on a schedule based on a select query (feature is still experimental):
https://clickhouse.com/docs/en/sql-reference/statements/create/view#refreshable-materialized-view