Skip to content

YETUS-1059. Github Status Recovery Tool #184

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
aw-was-here merged 2 commits into from
Nov 2, 2020
Merged

YETUS-1059. Github Status Recovery Tool #184

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
57da926
YETUS-1059. Github Status Recovery Tool
aw-was-here Oct 30, 2020
1df12da
fixup! YETUS-1059. Github Status Recovery Tool
aw-was-here Oct 31, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 25 additions & 3 deletions Jenkinsfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,9 +104,6 @@ pipeline {
YETUS_ARGS+=(--jira-password="${JIRA_PASSWORD}")
YETUS_ARGS+=(--jira-user="${JIRA_USER}")

# disable per-line comments
YETUS_ARGS+=(--linecomments='')

# pylint settings
YETUS_ARGS+=('--pylint=pylint2')

Expand Down Expand Up @@ -167,6 +164,31 @@ pipeline {
post {
always {
script {

// Publish status if it was missed
withCredentials([usernamePassword(credentialsId: 'apache-yetus-at-github.com',
passwordVariable: 'GITHUB_TOKEN',
usernameVariable: 'GITHUB_USER')]) {
sh '''#!/usr/bin/env bash

# enable writing back to Github
YETUS_ARGS+=(--github-token="${GITHUB_TOKEN}")
YETUS_ARGS+=("--patch-dir=${WORKSPACE}/${YETUS_RELATIVE_PATCHDIR}")

if [[ "${USE_DEBUG_FLAG}" == true ]]; then
YETUS_ARGS+=("--debug")
fi

# run test-patch from the source tree specified up above
TESTPATCHBIN=${WORKSPACE}/src/precommit/src/main/shell/github-status-recovery.sh

# execute! (we are using bash instead of the
# bin in case the perms get messed up)
/usr/bin/env bash "${TESTPATCHBIN}" "${YETUS_ARGS[@]}" ${EXTRA_ARGS} || true

'''
}

// Publish JUnit results
try {
junit "${env.YETUS_RELATIVE_PATCHDIR}/junit-report.xml"
Expand Down
50 changes: 50 additions & 0 deletions ...e-src/source/documentation/in-progress/precommit/github-status-recovery.html.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
<!---
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

# GitHub Status Recovery

<!-- MarkdownTOC levels="1,2,3" autolink="true" indent=" " bullets="*" bracket="round" -->

* [Problem Statement](#problem-statement)
* [Usage](#usage)
* [Disabling Annotations](#disabling-annotations)

<!-- /MarkdownTOC -->

# Problem Statement

For CI systems that use GitHub outside of GitHub Actions, they may make available a GitHub Checks token.
Unfortunately, as of this writing (2020-10-30), GitHub sets the expiry of such a token to 1 hour.
For some users of Apache Yetus, their precommit job may take longer than one hour. In order to workaround
this limitation, the `github-status-recovery` program may be used.

# Usage

The usage is relatively simple:

```bash
$ github-status-recovery --patch-dir=<pre-existing patch directory> --github-token=<token>
```

If the previous run of `test-patch` failed to write the status, `github-status-recovery` will
re-process the saved JSON files as well as write GitHub Checks Annotations if they exist.

# Disabling Annotations

If for some reason you do not wish annotations to be written, they may be disabled with `--github-annotations=false`.
1 change: 1 addition & 0 deletions asf-site-src/source/documentation/in-progress/precommit/index.html.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -156,6 +156,7 @@ Language Support, Licensing, and more:
capacities without needing to use the full `test-patch` runtime:

* [docker-cleanup](docker-cleanup) - safe removal of Docker resources for multi-executor CI systems
* [github-status-recovery](github-status-recovery) - Apache Yetus status on GitHub, even for long running jobs
* [jenkins-admin](admin) - Jenkins<->JIRA patch bridge
* [qbt](qbt) - Quality Build Tool, for branch-specific testing
* [smart-apply-patch](smart-apply-patch) - CLI manipulation and query of patch files, PRs, and more
Expand Down
23 changes: 21 additions & 2 deletions asf-site-src/source/documentation/in-progress/precommit/robots/jenkins.html.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,8 +90,27 @@ for the optimal `test-patch` experience. Configure up to the "Configuring the G
...
```

Doing so will enable in many circumstances a bit more functionality, such as
GitHub Statuses.
Doing so will enable in many circumstances a bit more functionality, such as GitHub Statuses.

## Enabling GitHub Status Recovery

If the Apache Yetus section of your job typically runs longer than 1 hour and you use GitHub as the primary bugsystem,
it is recommend to use the `github-status-recovery` utility in the `post` section of your `Jenkinsfile`. For example:

```groovy
post {
always {
script {
// Publish status if it was missed
withCredentials([usernamePassword(credentialsId: github-app',
passwordVariable: 'GITHUB_TOKEN',
usernameVariable: 'GITHUB_USER')]) {
sh '''github-status-recovery --patch-dir="${PATCH_DIR}"" --github-token="${GITHUB_TOKEN}"'''
}
}
}
}
```

See also:

Expand Down
132 changes: 132 additions & 0 deletions precommit/src/main/shell/core.d/linecomments.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,132 @@
#!/usr/bin/env bash
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Make sure that bash version meets the pre-requisite

if [[ -z "${BASH_VERSINFO[0]}" ]] \
|| [[ "${BASH_VERSINFO[0]}" -lt 3 ]] \
|| [[ "${BASH_VERSINFO[0]}" -eq 3 && "${BASH_VERSINFO[1]}" -lt 2 ]]; then
echo "bash v3.2+ is required. Sorry."
exit 1
fi

## @description Queue up comments to write into bug systems
## @description that have code review support, if such support
## @description enabled/available.
## @description File should be in the form of "file:line[:column]:comment"
## @audience public
## @stability evolving
## @replaceable no
## @param plugin
## @param filename
function bugsystem_linecomments_queue
{
declare plugin=$1
declare fn=$2
declare line
declare linenum
declare text
declare columncheck
declare column
declare rol
declare file

if [[ -z "${BUGLINECOMMENTS}" ]]; then
return 0
fi

for line in "${VOTE_FILTER[@]}"; do
if [[ "${plugin}" == "${line}" ]]; then
return 0
fi
done

pushd "${BASEDIR}" >/dev/null || return 1
while read -r line; do
file=${line%%:*}

if [[ ! -e "${file}" ]]; then
continue
fi

rol=${line/#${file}:}
if [[ "${file}" =~ ^\./ ]]; then
file=${file:2}
fi

linenum=${rol%%:*}
rol=${rol/#${linenum}:}
columncheck=${rol%%:*}
if [[ "${columncheck}" =~ ^[0-9]+$ ]]; then
column=${columncheck}
text=${rol/#${column}:}
else
column="0"
text=${rol}
fi

echo "${file}:${linenum}:${column}:${plugin}:${text}" >> "${PATCH_DIR}/results-full.txt"

done < "${fn}"

popd >/dev/null || return 1

}

## @description Write all of the bugsystem linecomments
## @audience public
## @stability evolving
## @replaceable no
function bugsystem_linecomments_trigger
{
declare plugin
declare fn
declare line
declare linenum
declare text
declare column

if [[ ! -f "${PATCH_DIR}/results-full.txt" ]]; then
return 0
fi

# sort the file such that all files and lines are now next to each other
sort -k1,1 -k2,2n -k3,3n -k4,4 "${PATCH_DIR}/results-full.txt" > "${PATCH_DIR}/linecomments-sorted.txt"
mv "${PATCH_DIR}/linecomments-sorted.txt" "${PATCH_DIR}/results-full.txt"

while read -r line;do
fn=${line%%:*}
rol=${line/#${fn}:}
linenum=${rol%%:*}
rol=${rol/#${linenum}:}
column=${rol%%:*}
rol=${rol/#${column}:}
plugin=${rol%%:*}
text=${rol/#${plugin}:}

for bugs in ${BUGLINECOMMENTS}; do
if declare -f "${bugs}_linecomments" >/dev/null;then
"${bugs}_linecomments" "${fn}" "${linenum}" "${column}" "${plugin}" "${text}"
fi
done
done < "${PATCH_DIR}/results-full.txt"

for bugs in ${BUGLINECOMMENTS}; do
if declare -f "${bugs}_linecomments_end" >/dev/null;then
"${bugs}_linecomments_end"
fi
done
}
Loading