Skip to content
This repository has been archived by the owner on Jun 6, 2024. It is now read-only.

New RestServer Architecture: RestServer -> DB -> ApiServer #4651

Open
20 of 32 tasks
yqwang-ms opened this issue Jun 28, 2020 · 1 comment
Open
20 of 32 tasks

New RestServer Architecture: RestServer -> DB -> ApiServer #4651

yqwang-ms opened this issue Jun 28, 2020 · 1 comment
Assignees
Labels

Comments

@yqwang-ms
Copy link
Member

yqwang-ms commented Jun 28, 2020

By leveraging DB, RestServer can be

  1. RAW Consistency
  2. High Perf and Powerful Query: List, Paging, Sorting, Summarizing, etc
  3. Larger storage quota and duration
  4. Active and history jobs are unified and merged together naturally (no need to introduce UID) (P0 - Job Name, UID and Description #3935)
  5. Job Name can submit idempotently, attach metadata arbitrarily, and query uniquely (P0 - Job Name, UID and Description #3935)
  6. etc

Features depend on it
List History Jobs: #3845, #4610, #3935
Expose K8s events: part of enrich job debugging info: #4649

New RestServer Architecture
In short, compared with current architecture, we insert a DB between RestServer and ApiServer.
image

Sub Tasks

  • Database Controller (StatefulSet with initializer, write merger, watcher, poller, gcer)
    • P0 Database schema
    • P0 Database ORM interface in Node.js
    • P0 Service setup (sevice.yaml, start.sh, config, ... , etc)
    • Initializer (handle schema sync, upgrade, legacy, etc...)
      • P0 Version table setup
      • P1 Legacy framework transfer @suiguoxin
    • P0 Write merger
    • P0.5 DB poller
    • API watcher
      • P0 Watcher for framework
      • P2 Watcher for event
      • P2 Watcher for pod
    • P1 DB GCer #5653
  • Rest API Change
    • P0 POST /api/v2/jobs
    • P0 GET /api/v2/jobs
    • P0 GET /api/v2/jobs/{user}~{job}
    • P0 GET /api/v2/jobs/{user}~{job}/config
    • P0 PUT /api/v2/jobs/{user}~{job}/executionType
    • P0 Swagger update
    • P1 /api/v2/jobs/{user}~{job}/job-attempts @suiguoxin Db migration #4716
    • P1 /api/v2/jobs/{user}~{job}/job-attempts/{attemptIndex} @suiguoxin
  • Webportal
    • P1 paging on list job page
  • Fluentd @suiguoxin Db migration #4716
    • P1 Fluentd plugin multithread
    • P1 Fluentd change for new schema (for framework history)
    • P2 Fluentd for Pod
  • Other
    • P1 Framework controller Decrease GC time?
    • P2 Make postgresql a StatefulSet
@yqwang-ms yqwang-ms changed the title [DESIGN DOC]: New RestServer Architecture: RestServer -> DB -> ApiServer New RestServer Architecture: RestServer -> DB -> ApiServer Jun 28, 2020
@hzy46 hzy46 mentioned this issue Jun 28, 2020
21 tasks
@fanyangCS
Copy link
Contributor

relate to #4600

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

No branches or pull requests

4 participants