Skip to content
Permalink
Browse files
Support fetch released docs from main repo (#775)
* Support fetch released docs from main repo

All our history docs before we migrate doc to apache/dolphinscheduler
is in apache/dolphinscheduler-website branch `history-docs`.

After we migrate doc to apache/dolphinscheduler we have two ways to
handle its release docs, one of the ways is copy-paste docs from
apache/dolphinscheduler to apache/dolphinscheduler-website branch
`history-docs`, the other way is to find some black magic fetch docs
based on the release tag.

Of cause, we prefer to choose the second way. Fetch base on released
tags.

* rename bash script from release.conf.sh to conf.sh

* Change fetch branch instead of tag

* fix
  • Loading branch information
zhongjiajie committed Apr 18, 2022
1 parent 5e5de55 commit 5b5725fb9e4cee6056c1bcfbfc2d1ff7fee249ce
Show file tree
Hide file tree
Showing 5 changed files with 233 additions and 81 deletions.
@@ -22,7 +22,7 @@ on:
- master

env:
DOCS_FLAG: ^docs
DOCS_FLAG: ^docs|^development

concurrency:
group: docs-${{ github.event.pull_request.number || github.ref }}
@@ -41,7 +41,7 @@ jobs:
- name: Get Branch origin/dev
run: |
git fetch --no-tags --prune --depth=1 origin master
- name: Fail When Docs Dir Change
- name: Fail when change directory `docs` or `development`
run: |
if git diff --name-only origin/master HEAD | grep -q -E "${{ env.DOCS_FLAG }}"; then
exit 1
@@ -7,16 +7,48 @@ and the history documents before version 3.0.0-alpha is in branch [history-docs]
in [website repository](https://github.com/apache/dolphinscheduler-website). In this case, you have to collect them from
above repository into current working path before you compile them to HTML.

## Execute Command and Prepare Resource

Of course, you could collect all content manually, but we already provider convenience script to do that, all you have to
do is run a single command in this project root directory:

```shell
./scripts/prepare_docs.sh
```

It would do all prepare things for you.
> Note: The default protocol to fetch repository from GitHub is via HTTPS. When you failed to run the command above and
> see some log like "unable to access" in your terminal, or if you feel clone via HTTPS is a little slow. You can switch
> clone protocol to SSH by change the environment variable by execute command `export PROTOCOL_MODE=ssh` in your terminal.
> The next time you execute command `./scripts/prepare_docs.sh` will user protocol SSH instead of HTTPS, which is more stable
> and fast in some cases, such as local development
After the command finished, all prepare things you need to start DolphinScheduler website is being done.

## How To Add or Change New Release Document From apache/dolphinscheduler

After version 3.0.0-alpha, our release document is in repository [apache/dolphinscheduler](https://github.com/apache/dolphinscheduler)
directory `docs`. Documentation and code will be released together, so when the new version is released, we need to get
the corresponding documentation from the released tag.

All you have to do it is add/change variable `DEV_RELEASE_DOCS_VERSIONS` in [conf.sh](scripts/conf.sh).
For example, if you want to add new release docs named `10.0.0` and its docs in apache/dolphinscheduler branch `10.0.0-release`,
you could add a new key-value to `DEV_RELEASE_DOCS_VERSIONS`

```shell
declare -A DEV_RELEASE_DOCS_VERSIONS=(
...
["10.0.0"]="10.0.0-release"
...
)
```

A few days later, you find out version `10.0.0`'s document has a bug. You fix it and push it to apache/dolphinscheduler
branch `10.0.0-release` to fix the bug, you could keep your configuration in the old content.

> Note: When you failed to run the command and see some log like "unable to access" in your terminal, you can set and
> environment variable `export PROTOCOL_MODE=ssh` and then run command `./scripts/prepare_docs.sh` again. After setting the
> variable will clone source code in SSH protocol instead of HTTPS protocol, and it will stable and fast in some cases like
> local development.
```shell
declare -A DEV_RELEASE_DOCS_VERSIONS=(
...
["10.0.0"]="10.0.0-release"
...
)
```
@@ -0,0 +1,45 @@
#!/usr/bin/env bash

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

# Codebase Repository
PROJECT_NAME="dolphinscheduler"
PROJECT_BRANCH_NAME="dev"
PROJECT_WEBSITE_NAME="${PROJECT_NAME}-website"
PROJECT_WEBSITE_BRANCH_NAME="history-docs"

# Repository Website(current) Directory And Files
SWAP_DIR="${SOURCE_PATH}/swap"
PROJECT_SITE_DOC_DIR="${SOURCE_PATH}/docs"
PROJECT_SITE_DEVELOP_DIR="${SOURCE_PATH}/development"
PROJECT_DIR="${SWAP_DIR}/${PROJECT_NAME}"
PROJECT_WEBSITE_DIR="${SWAP_DIR}/${PROJECT_WEBSITE_NAME}"

# docs in apache/dolphinscheduler-webstie branch `history-docs`
declare -a HISTORY_DOCS_VERSIONS=(
"1.2.0" "1.2.1" "1.3.1" "1.3.2" "1.3.3" "1.3.4" "1.3.5" "1.3.6" "1.3.8" "1.3.9"
"2.0.0" "2.0.1" "2.0.2" "2.0.3" "2.0.5"
)

# docs in apache/dolphinscheduler directory `docs` after 3.0.0-alpha(the time we migrate docs to this repo)
declare -A DEV_RELEASE_DOCS_VERSIONS=(
# The key value is represents of version `key` document is from tag `val` from apache/dolphinscheduler
# example: ["key"]="val"
["3.0.0"]="3.0.0-alpha-release"
)

@@ -21,24 +21,7 @@ set -euo pipefail

SOURCE_PATH="$(cd "$(dirname "$(dirname "${BASH_SOURCE[0]}")" )" && pwd)"

# Codebase Repository
PROJECT_NAME="dolphinscheduler"
PROJECT_BRANCH_NAME="dev"
PROJECT_WEBSITE_NAME="${PROJECT_NAME}-website"
PROJECT_WEBSITE_BRANCH_NAME="history-docs"

# Repository Website(current) Directory And Files
SWAP_DIR="${SOURCE_PATH}/swap"
PROJECT_SITE_DOC_DIR="${SOURCE_PATH}/docs"
PROJECT_SITE_DEVELOP_DIR="${SOURCE_PATH}/development"
PROJECT_DIR="${SWAP_DIR}/${PROJECT_NAME}"
PROJECT_WEBSITE_DIR="${SWAP_DIR}/${PROJECT_WEBSITE_NAME}"

declare -a versions=(
"1.2.0" "1.2.1" "1.3.1" "1.3.2" "1.3.3" "1.3.4" "1.3.5" "1.3.6" "1.3.8" "1.3.9"
"2.0.0" "2.0.1" "2.0.2" "2.0.3" "2.0.5"
)

source "${SOURCE_PATH}/scripts/conf.sh"

# Choose the protocol for git communication to server, default is HTTP because it do not requests password or secret key,
# run command `export PROTOCOL_MODE=ssh` in terminal change protocol to SSH which in is faster and stable in many cases,
@@ -72,37 +55,6 @@ function rebuild_dirs() {
done
}

##############################################################
#
# Sync command wrapper about copy from content from src to
# dest. It will create a directory for dest to make sure the
# rsync, and support rsync options.
#
# Arguments:
#
# src: rsync source parameter
# dest: rsync destent parameter
# rsync options: rsync command options
#
##############################################################
function rsync_wrapper() {
local src="${1}"
local dest="${2}"
local dir_flag=""
# Create directory
if [ -d "${src}" ]; then
mkdir -p "${dest}"
dir_flag="/"
else
mkdir -p "$(dirname "${dest}")"
fi
if [ "$#" -eq 2 ]; then
rsync -av ${src}${dir_flag} "${dest}"
else
rsync -av ${3} ${src}${dir_flag} "${dest}"
fi
}

##############################################################
#
# Clone repository to target directory, it will only support
@@ -185,33 +137,16 @@ function prepare_docs() {
echo " ---> Clone dev documents from ${PROJECT_REPO} branch ${PROJECT_BRANCH_NAME}."
clone_repo "${PROJECT_REPO}" "${PROJECT_BRANCH_NAME}" "${PROJECT_DIR}"

source "${SOURCE_PATH}/scripts/rsync_content.sh"
echo "===>>> Sync content from cloned repository."
echo " ---> Sync history content."
for version in "${versions[@]}"; do
echo " --> Sync version ${version} content"
rsync_wrapper "${PROJECT_WEBSITE_DIR}/docs/${version}/docs/en" "${PROJECT_SITE_DOC_DIR}/en-us/${version}/user_doc"
rsync_wrapper "${PROJECT_WEBSITE_DIR}/docs/${version}/docs/zh" "${PROJECT_SITE_DOC_DIR}/zh-cn/${version}/user_doc"
echo " ---> Sync version ${version} configs."
local config_file_stem="docs${version//./-}"
rsync_wrapper "${PROJECT_WEBSITE_DIR}/docs/${version}/configs/${config_file_stem}.js" "${SOURCE_PATH}/site_config/${config_file_stem}.js"
done
echo " ---> Sync history img."
rsync_wrapper "${PROJECT_WEBSITE_DIR}/img"/ "${SOURCE_PATH}/img"

echo " ---> Sync dev user document content."
rsync_wrapper "${PROJECT_DIR}/docs/docs/en" "${PROJECT_SITE_DOC_DIR}/en-us/dev/user_doc" "--exclude=faq.md --exclude=history-versions.md --exclude=development"
rsync_wrapper "${PROJECT_DIR}/docs/docs/en/*.md" "${PROJECT_SITE_DOC_DIR}/en-us/release"
rsync_wrapper "${PROJECT_DIR}/docs/docs/zh" "${PROJECT_SITE_DOC_DIR}/zh-cn/dev/user_doc" "--exclude=faq.md --exclude=history-versions.md --exclude=development"
rsync_wrapper "${PROJECT_DIR}/docs/docs/zh/*.md" "${PROJECT_SITE_DOC_DIR}/zh-cn/release"
echo " ---> Sync development document content."
rsync_wrapper "${PROJECT_DIR}/docs/docs/en/development" "${PROJECT_SITE_DEVELOP_DIR}/en-us"
rsync_wrapper "${PROJECT_DIR}/docs/docs/zh/development" "${PROJECT_SITE_DEVELOP_DIR}/zh-cn"
echo " ---> Sync dev img."
rsync_wrapper "${PROJECT_DIR}/docs/img" "${SOURCE_PATH}/img"
echo " ---> Sync dev configs."
rsync_wrapper "${PROJECT_DIR}/docs/configs/site.js" "${SOURCE_PATH}/site_config/site.js"
rsync_wrapper "${PROJECT_DIR}/docs/configs/index.md.jsx" "${SOURCE_PATH}/src/pages/docs/index.md.jsx"
rsync_wrapper "${PROJECT_DIR}/docs/configs/docsdev.js" "${SOURCE_PATH}/site_config/docsdev.js"
rsync_history_docs

echo " ---> Sync released document after migrating docs into apache/dolphinscheduler."
rsync_released_docs

echo " ---> Sync latest document in dev branch."
rsync_latest_docs

echo "===>>> End prepare document."
}
@@ -0,0 +1,140 @@
#!/usr/bin/env bash

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

set -euo pipefail

SOURCE_PATH="$(cd "$(dirname "$(dirname "${BASH_SOURCE[0]}")" )" && pwd)"

source "${SOURCE_PATH}/scripts/conf.sh"

##############################################################
#
# Sync command wrapper about copy from content from src to
# dest. It will create a directory for dest to make sure the
# rsync, and support rsync options.
#
# Arguments:
#
# src: rsync source parameter
# dest: rsync destent parameter
# rsync options: rsync command options
#
##############################################################
function rsync_wrapper() {
local src="${1}"
local dest="${2}"
local dir_flag=""
# Create directory
if [ -d "${src}" ]; then
mkdir -p "${dest}"
dir_flag="/"
else
mkdir -p "$(dirname "${dest}")"
fi
if [ "$#" -eq 2 ]; then
rsync -av ${src}${dir_flag} "${dest}"
else
rsync -av ${3} ${src}${dir_flag} "${dest}"
fi
}

##############################################################
#
# Rsync history document, from cloned repo
# apache/dolphinscheduler-website branch `history-docs`, to
# directory `docs` and `img`.
#
##############################################################
function rsync_history_docs() {
for version in "${HISTORY_DOCS_VERSIONS[@]}"; do
echo " --> Sync version ${version} content"
rsync_wrapper "${PROJECT_WEBSITE_DIR}/docs/${version}/docs/en" "${PROJECT_SITE_DOC_DIR}/en-us/${version}/user_doc"
rsync_wrapper "${PROJECT_WEBSITE_DIR}/docs/${version}/docs/zh" "${PROJECT_SITE_DOC_DIR}/zh-cn/${version}/user_doc"

echo " ---> Sync version ${version} configs."
local config_file_stem="docs${version//./-}"
rsync_wrapper "${PROJECT_WEBSITE_DIR}/docs/${version}/configs/${config_file_stem}.js" "${SOURCE_PATH}/site_config/${config_file_stem}.js"
done

echo " ---> Sync history img in batch mode."
rsync_wrapper "${PROJECT_WEBSITE_DIR}/img"/ "${SOURCE_PATH}/img"
}

##############################################################
#
# Rsync released document, from cloned repo
# apache/dolphinscheduler tags after we migrate docs to the
# main repo(after 3.0.0-alpha), to directory `docs`, `img` and
# `site_config`.
#
##############################################################
function rsync_released_docs() {
echo " ---> Directory change to ${PROJECT_DIR}."
cd "${PROJECT_DIR}"

for version in "${!DEV_RELEASE_DOCS_VERSIONS[@]}"; do
echo " ---> Git checkout to version ${version}."
git fetch origin "${DEV_RELEASE_DOCS_VERSIONS[$version]}" --no-tags
git checkout -b "${version}" FETCH_HEAD

echo " ---> Sync released version ${version} docs."
rsync_wrapper "${PROJECT_DIR}/docs/docs/en" "${PROJECT_SITE_DOC_DIR}/en-us/${version}/user_doc" "--exclude=faq.md --exclude=history-versions.md --exclude=development"
rsync_wrapper "${PROJECT_DIR}/docs/docs/zh" "${PROJECT_SITE_DOC_DIR}/zh-cn/${version}/user_doc" "--exclude=faq.md --exclude=history-versions.md --exclude=development"

echo " ---> Sync released version ${version} img."
rsync_wrapper "${PROJECT_DIR}/docs/img" "${SOURCE_PATH}/img"
echo " ---> Sync released version ${version} configs."

local config_file_stem="docs${version//./-}"
# To avoid change file `docsdev.js` name each time before release, we change it during our sync process
rsync_wrapper "${PROJECT_DIR}/docs/configs/docsdev.js" "${SOURCE_PATH}/site_config/${config_file_stem}.js"
done
}

##############################################################
#
# Rsync released document, from cloned repo
# apache/dolphinscheduler branch `dev`(latest), to directory
# `docs`, `img`, also with rsync config files.
#
##############################################################
function rsync_latest_docs() {
echo " ---> Directory change to ${PROJECT_DIR}."
cd "${PROJECT_DIR}"
echo " ---> Git checkout to version ${PROJECT_BRANCH_NAME}."
git checkout "${PROJECT_BRANCH_NAME}"

echo " ---> Sync dev docs."
rsync_wrapper "${PROJECT_DIR}/docs/docs/en" "${PROJECT_SITE_DOC_DIR}/en-us/dev/user_doc" "--exclude=faq.md --exclude=history-versions.md --exclude=development"
rsync_wrapper "${PROJECT_DIR}/docs/docs/en/*.md" "${PROJECT_SITE_DOC_DIR}/en-us/release"
rsync_wrapper "${PROJECT_DIR}/docs/docs/zh" "${PROJECT_SITE_DOC_DIR}/zh-cn/dev/user_doc" "--exclude=faq.md --exclude=history-versions.md --exclude=development"
rsync_wrapper "${PROJECT_DIR}/docs/docs/zh/*.md" "${PROJECT_SITE_DOC_DIR}/zh-cn/release"

echo " ---> Sync development document content."
rsync_wrapper "${PROJECT_DIR}/docs/docs/en/development" "${PROJECT_SITE_DEVELOP_DIR}/en-us"
rsync_wrapper "${PROJECT_DIR}/docs/docs/zh/development" "${PROJECT_SITE_DEVELOP_DIR}/zh-cn"

echo " ---> Sync dev img."
rsync_wrapper "${PROJECT_DIR}/docs/img" "${SOURCE_PATH}/img"

echo " ---> Sync dev configs."
rsync_wrapper "${PROJECT_DIR}/docs/configs/site.js" "${SOURCE_PATH}/site_config/site.js"
rsync_wrapper "${PROJECT_DIR}/docs/configs/index.md.jsx" "${SOURCE_PATH}/src/pages/docs/index.md.jsx"
rsync_wrapper "${PROJECT_DIR}/docs/configs/docsdev.js" "${SOURCE_PATH}/site_config/docsdev.js"
}

0 comments on commit 5b5725f

Please sign in to comment.