Docker & K8s Improvement Plan

[chengshiwen]

1.3.5-release

Docker Improvement Plan

• Reduce image size
• Remove frontend/nginx
• Remove stdout log in properties
• improve compose/swarm deployment
• Fix random PSQLException
• Sync docker conf templates to the latest conf properties
• Update outdated readme

K8s Improvement Plan

• Remove frontend/nginx
• Remove redundant configs
• Support resource storage when fs.defaultFS=file:///
• Fix task log disappear after recreating or upgrading helm release
• Improve long host display in K8s
• Update outdated readme

dev & 1.3.6-release

Docker Improvement Plan

• Reduce image size
• Remove frontend/nginx
• Remove stdout log in properties
• improve compose/swarm deployment
• Fix random PSQLException
• Sync docker conf templates to the latest conf properties
• Update outdated readme
• Rename WORKER_GROUP to WORKER_GROUPS ([Bug][Docker] 1.3.5 release compose file config error WORKER_GROUP #4843)
• Add FAQ support about MySQL and Oracle in readme ([Question] How to solve the problem of using source mode docker to build project and create Oracle data source failure #4732 [Question] Unable to Connect to MySQL datasource #4871)
• Add default login username and password in readme ([Improvement][Docker] Add default login username and password in readme #4881)
• Check compatibility in Windows ([BUG][DOCKER] docker compose doesn't work #4804)
• Fix invalid volume specification in Windows ([Bug][Docker] Cannot create container for service dolphinscheduler-worker: invalid volume specification in Windows #4886)
• Support supervisor management ([Feature]use supervisor(or other) to manage the daemon processes in docker #2687)
• Optimize log stdout with supervisor ([Bug][Docker] docker logs -f command cannot keep outputting logs all the time #4897)
• Fix that can not build a docker image on Windows ([BUG] a error when build a docker image #3124)
• Support docker compose file 3.1+ version
• Add tenant tip for quick start
• Sync the latest configs in master.properties and worker.properties
• Support all configs in common.properties
• Improve the environment variables and reduce the duplication configs
• Fix mysql check
• Add config.env.sh for docker compose and swarm
• Add MASTER_SERVER_OPTS, WORKER_SERVER_OPTS, ALERT_SERVER_OPTS, API_SERVER_OPTS, LOGGER_SERVER_OPTS
• Support skywalking config in docker
• Replace alpine with debian:slim (A Linux distribution for docker-compose #5155)
• Remove redundant postgresql-client and mysql-client
• Add faq for python, spark, hadoop, flink and datax
• Add docs for support matrix on docker ([Improvement][Docker/K8s] Task support matrix and solution on docker/k8s #5068)
• Support task scalability on Docker such as Spark, Flink and DataX ([Feature][K8s] Support task scalability on K8s such as Spark, Flink and DataX #5100)

K8s Improvement Plan

• Remove frontend/nginx
• Remove redundant configs
• Support resource storage when fs.defaultFS=file:///
• Fix task log disappear after recreating or upgrading helm release
• Improve long host display in K8s
• Adapt to the latest alert-server with updated config and port 50052
• Update outdated readme
• Rename WORKER_GROUP to WORKER_GROUPS ([Bug][Docker] 1.3.5 release compose file config error WORKER_GROUP #4843)
• Add FAQ support about MySQL and Oracle in readme ([Feature] 建议k8s部署的时候默认数据库是MySQL，降低使用门槛 #2584)
• Add FAQ support to use namespace, access ui and delete pv in readme
• Incorrect host problem in minikube ([Bug][K8s] Incorrect host problem in minikube #4905)
• Check compatibility in all Kubernetes 1.12+
• Support local docker image
• Optimize image pull secret
• Optimize fullname
• Optimize metadata name
• Sync the latest configs in master.properties and worker.properties
• Support all configs in common.properties
• Improve the environment variables and reduce the duplication configs
• Support skywalking config in k8s
• Unify annotations, affinity, nodeSelector, tolerations, resources and probe in K8s
• Support more service access like ClusterIP, NodePort and LoadBalancer
• Improve https tip in k8s
• Add MASTER_SERVER_OPTS, WORKER_SERVER_OPTS, ALERT_SERVER_OPTS, API_SERVER_OPTS, LOGGER_SERVER_OPTS
• Add faq for python, spark, hadoop, flink and datax
• Add docs for support matrix on k8s ([Improvement][Docker/K8s] Task support matrix and solution on docker/k8s #5068)
• Support task scalability on K8s such as Spark, Flink and DataX ([Feature][K8s] Support task scalability on K8s such as Spark, Flink and DataX #5100)
• Fix quartz cron task cannot take effect in K8s ([Bug][K8s] Quartz cron task cannot take effect #5351)
• Reduce long host and improve long host display on K8s

[chengshiwen]

@dailidong @chenxingchun

[xingchun-chen]

good job

[davidzollo]

good job