-
Notifications
You must be signed in to change notification settings - Fork 185
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[FLINK-16260] Change generate-stackbrew-library.sh to support new release.metadata file #31
Closed
Closed
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,6 +1,6 @@ | ||
#!/bin/bash | ||
#!/usr/bin/env bash | ||
|
||
exec docker run --rm \ | ||
--volume "${PWD}:/build:ro" \ | ||
plucas/docker-flink-build \ | ||
rmetzger/git-and-bash:latest \ | ||
/build/generate-stackbrew-library.sh |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,4 +1,4 @@ | ||
#!/bin/bash | ||
#!/usr/bin/env bash | ||
|
||
# This script generates a manifest compatibile with the expectations set forth | ||
# by docker-library/official-images. | ||
|
@@ -8,16 +8,9 @@ | |
|
||
set -eu | ||
|
||
declare -A aliases=( | ||
[1.11]='latest' | ||
) | ||
|
||
self="$(basename "$BASH_SOURCE")" | ||
cd "$(dirname "$(readlink -f "$BASH_SOURCE")")" | ||
|
||
# Identify directories matching '?.?' (e.g. '1.7') and remove trailing slashes | ||
versions=( ?.?/ ?.??/ ) | ||
versions=( "${versions[@]%/}" ) | ||
|
||
# get the most recent commit which modified any of "$@" | ||
fileCommit() { | ||
|
@@ -41,21 +34,32 @@ dirCommit() { | |
) | ||
} | ||
|
||
getArches() { | ||
local repo="$1"; shift | ||
local officialImagesUrl='https://github.com/docker-library/official-images/raw/master/library/' | ||
# Inputs: | ||
# - tags: comma-seprated list of image tags | ||
# - latestVersion: latest version | ||
# Output: comma-separated list of tags with "latest" removed if not latest version | ||
pruneTags() { | ||
local inTagsString=$1 | ||
local latestVersion=$2 | ||
if [[ $inTagsString =~ $latestVersion ]]; then | ||
# tagsString contains latest version. keep "latest" tag | ||
echo $inTagsString | ||
else | ||
# remove "latest" and any "scala_" tag, unless it is the latest version | ||
# the "scala_" tag has a similar semantic as the "latest" tag in docker registries. | ||
echo $inTagsString | sed -E 's|,(scala\|latest)[-_[:alnum:]]*||g' | ||
fi | ||
} | ||
|
||
eval "declare -g -A parentRepoToArches=( $( | ||
find -name 'Dockerfile' -exec awk ' | ||
toupper($1) == "FROM" && $2 !~ /^('"$repo"'|scratch|microsoft\/[^:]+)(:|$)/ { | ||
print "'"$officialImagesUrl"'" $2 | ||
} | ||
' '{}' + \ | ||
| sort -u \ | ||
| xargs bashbrew cat --format '[{{ .RepoName }}:{{ .TagName }}]="{{ join " " .TagEntry.Architectures }}"' | ||
) )" | ||
extractValue() { | ||
local key="$1" | ||
local file="$2" | ||
local line=$(cat $file | grep "$key:") | ||
echo $line | cut -d ':' -f 2 | tr -d ' ' # remove key from line, remove whitespace | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I looked at various entries in https://github.com/docker-library/official-images and they all separated tags with a comma and space We could use |
||
} | ||
getArches 'flink' | ||
|
||
# get latest flink version | ||
latest_version=`ls -1a | grep -E "[0-9]+.[0-9]+" | sort -V -r | head -n 1` | ||
|
||
cat <<-EOH | ||
# this file is generated via https://github.com/apache/flink-docker/blob/$(fileCommit "$self")/$self | ||
|
@@ -65,105 +69,25 @@ Maintainers: Patrick Lucas <me@patricklucas.com> (@patricklucas), | |
GitRepo: https://github.com/apache/flink-docker.git | ||
EOH | ||
|
||
# prints "$2$1$3$1...$N" | ||
join() { | ||
local sep="$1"; shift | ||
local out; printf -v out "${sep//%/%%}%s" "$@" | ||
echo "${out#$sep}" | ||
} | ||
|
||
# Sorry for the style here, but it makes the nested code easier to read | ||
for version in "${versions[@]}"; do | ||
|
||
# Defaults, can vary between versions | ||
source_variants=( debian ) | ||
scala_versions=( 2.11 2.12 ) | ||
|
||
# Version-specific variants (example) | ||
# if [ "$flink_release" = "x.y" ]; then | ||
# scala_versions=( 2.10 2.11 2.12 ) | ||
# fi | ||
|
||
for source_variant in "${source_variants[@]}"; do | ||
for scala_version in "${scala_versions[@]}"; do | ||
dir="$version/scala_${scala_version}-${source_variant}" | ||
|
||
# Not all variant combinations may exist | ||
[ -f "$dir/Dockerfile" ] || continue | ||
for dockerfile in $(find . -name "Dockerfile"); do | ||
dir=$(dirname $dockerfile) | ||
|
||
commit="$(dirCommit "$dir")" | ||
metadata="$dir/release.metadata" | ||
architectures=$(extractValue "Architectures" $metadata) | ||
tags=$(extractValue "Tags" $metadata) | ||
tags=$(pruneTags "$tags" $latest_version) | ||
|
||
# Extract the full Flink version from the Dockerfile | ||
flink_version="$(git show "$commit":"$dir/Dockerfile" | grep 'dist/flink/flink-[0-9.]*' | cut -d/ -f6 | cut -d- -f2)" | ||
|
||
full_version=$flink_version-scala_$scala_version | ||
|
||
variantParent="$(awk 'toupper($1) == "FROM" { print $2 }' "$dir/Dockerfile")" | ||
variantArches="${parentRepoToArches[$variantParent]}" | ||
|
||
# Start with the full version e.g. "1.2.0-scala_2.11" and add | ||
# additional tags as relevant | ||
tags=( $full_version ) | ||
|
||
is_latest_version= | ||
[ "$version" = "${versions[-1]}" ] && is_latest_version=1 | ||
|
||
is_latest_scala= | ||
[ "$scala_version" = "${scala_versions[-1]}" ] && is_latest_scala=1 | ||
|
||
add_tags=( $version ) | ||
|
||
# Add a scala version tag to each image | ||
tags=( | ||
${tags[@]} | ||
${add_tags[@]/%/-scala_$scala_version} | ||
) | ||
|
||
# If this is the latest Flink release, add a tag with only the scala version | ||
if [ -n "$is_latest_version" ]; then | ||
tags=( | ||
${tags[@]} | ||
"scala_$scala_version" | ||
) | ||
fi | ||
|
||
# For the latest supported Scala version, add tags that omit the scala version | ||
if [ -n "$is_latest_scala" ]; then | ||
tags=( | ||
${tags[@]} | ||
$flink_version | ||
$version | ||
) | ||
fi | ||
|
||
# Add -$variant suffix for non-debian-based images | ||
if [ "$source_variant" != "debian" ]; then | ||
tags=( ${tags[@]/%/-$source_variant} ) | ||
fi | ||
|
||
# Finally, designate the 'latest' tag (or '$variant', for non-debian-based images) | ||
if [ -n "$is_latest_scala" ]; then | ||
alias_tag="${aliases[$version]:-}" | ||
if [ -n "$alias_tag" ] && [ "$source_variant" != "debian" ]; then | ||
alias_tag="$source_variant" | ||
fi | ||
|
||
tags=( | ||
${tags[@]} | ||
$alias_tag | ||
) | ||
fi | ||
|
||
# newline | ||
echo | ||
|
||
# The tabs here are necessary for the heredoc to work right | ||
cat <<-EOE | ||
Tags: $(join ', ' "${tags[@]}") | ||
Architectures: $(join ', ' $variantArches) | ||
Tags: $tags | ||
Architectures: $architectures | ||
GitCommit: $commit | ||
Directory: $dir | ||
EOE | ||
|
||
done | ||
done | ||
done |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
variable name does not match (I'd use
tags
)