Skip to content

Align website with the v0.116 package matrix #164

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
guanzhousongmicrosoft merged 2 commits into from
Sep 1, 2026
Merged

Align website with the v0.116 package matrix #164

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
fdb3659
Align website with v0.116 package matrix
guanzhousongmicrosoft Aug 31, 2026
13c6099
Harden the v0.116 repository transition
guanzhousongmicrosoft Sep 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
198 changes: 92 additions & 106 deletions .github/scripts/download_packages.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,11 @@ set -e
REPO="documentdb/documentdb"
OUT_DIR="out"
DOCUMENTDB_VERSION="${DOCUMENTDB_VERSION:-latest}"
MULTI_VERSION="${MULTI_VERSION:-true}"
SUITE="${SUITE:-stable}"
COMPONENTS="${COMPONENTS:-main}"
ORIGIN="${ORIGIN:-DocumentDB}"
DESCRIPTION="${DESCRIPTION:-DocumentDB APT and YUM Repository}"
APT_METADATA_COMPONENTS="main deb11 deb12 deb13 ubuntu22 ubuntu24"

sign_deb_package() {
local package_file="$1"
Expand Down Expand Up @@ -76,34 +76,20 @@ echo "Downloading packages from $REPO releases"
# ---------------------------------------------------------------------------
# Release selection
#
# The primary release supplies the site's release-info.json and is the version
# users are told about. But a release only ships the distributions that were
# in its own build matrix: v0.116-0, for example, ships Tier-1 (ubuntu24 +
# rhel9) only, while v0.114-0 shipped seven distributions. Rebuilding the
# repository from the primary release alone would therefore DELETE the
# deb11/deb12/deb13/ubuntu22 components and the whole rhel8 repository from
# documentdb.io, and every host already pointed at one of them would start
# failing `apt update` with "Component 'ubuntu22' is not defined". That is a
# client-visible outage, not a cosmetic regression.
#
# So the repository is built additively: the primary release fills every
# distribution it ships, then progressively older releases are consulted ONLY
# to fill distributions still missing. A distribution is claimed by the newest
# release that ships it and is never overwritten by an older one. Once a
# release ships every distribution again, the older ones stop contributing
# by themselves - no cleanup required.
# The selected release is the package repository's single source of truth.
# Older releases must not fill gaps: those combinations are on-demand builds,
# not assets of the current official release, and mixing them into the pool
# makes stale versions look supported.
# ---------------------------------------------------------------------------
MAX_RELEASES="${MAX_RELEASES:-8}"

RELEASES_JSON=$(mktemp)
if ! curl -fqs "https://api.github.com/repos/${REPO}/releases?per_page=100" > "$RELEASES_JSON"; then
echo "Error: Could not fetch release list"
exit 1
fi

# Ordered list of tags to consider, newest first. Drafts and prereleases are
# skipped: they are not what a repository-backed `apt install` should serve.
TAG_LIST=$(DOCUMENTDB_VERSION="$DOCUMENTDB_VERSION" python3 - "$RELEASES_JSON" <<'PY'
# Select exactly one published release. Drafts and prereleases are skipped.
SELECTED_TAG=$(DOCUMENTDB_VERSION="$DOCUMENTDB_VERSION" python3 - "$RELEASES_JSON" <<'PY'
import json, os, sys

releases = json.load(open(sys.argv[1]))
Expand All @@ -113,33 +99,27 @@ if not published:

requested = os.environ.get("DOCUMENTDB_VERSION", "latest")
if requested != "latest":
# The API returns releases newest-first.
index = next((i for i, r in enumerate(published)
if r["tag_name"] == requested), None)
if index is None:
selected = next((r for r in published if r["tag_name"] == requested), None)
if selected is None:
sys.exit(f"Error: Version {requested} not found in releases")
# A pin means "serve this version". Only the pinned release and OLDER ones
# may contribute: pulling gap-fillers from NEWER releases would defeat the
# pin, and worse, it would mix releases that depend on each other. Pinning
# to a release that predates the multi-package layout would otherwise add a
# newer `documentdb` meta package whose `documentdb-N (>= X)` dependency the
# pinned extension cannot satisfy - an unsatisfiable repository.
ordered = published[index:]
else:
ordered = published
selected = published[0]

print("\n".join(r["tag_name"] for r in ordered))
print(selected["tag_name"])
PY
)

PRIMARY_TAG=$(printf '%s\n' "$TAG_LIST" | head -n 1)
PRIMARY_TAG="$SELECTED_TAG"
echo "Primary release: $PRIMARY_TAG"

# Packages already placed, keyed by "pool|name|arch". A newer release always
# wins; an older release contributes only packages the newer ones do not
# provide at all. That is what keeps `postgresql-16-documentdb` alive on
# ubuntu24/rhel9 after v0.116-0 narrowed Tier 1 to PostgreSQL 17/18, instead of
# silently deleting a package the install docs still tell people to use.
# The paragraph above describes the removed gap-fill implementation. The
# current implementation accepts assets only from PRIMARY_TAG and preserves
# retired URLs with empty metadata rather than stale packages.
SEEN_FILE=$(mktemp)

is_claimed() { case " $1 " in *" $2 "*) return 0 ;; *) return 1 ;; esac; }
Expand Down Expand Up @@ -223,13 +203,11 @@ rpm_pool_for() {

mkdir -p out/packages

for tag in $TAG_LIST; do
MAX_RELEASES=$((MAX_RELEASES - 1))
[ "$MAX_RELEASES" -lt 0 ] && break
for tag in "$PRIMARY_TAG"; do

if ! release=$(curl -fqs "https://api.github.com/repos/${REPO}/releases/tags/$tag"); then
echo "::warning::Could not fetch release $tag, skipping"
continue
echo "::error::Could not fetch selected release $tag"
exit 1
fi

ASSETS_FILE=$(mktemp)
Expand Down Expand Up @@ -263,7 +241,10 @@ for asset in data.get('assets', []):
comp=$(deb_component_for "$filename")
[ -z "$comp" ] && continue
claim_package "$comp" "$filename" || continue
wget -q -P out/packages "$download_url" || { echo "::warning::download failed: $filename"; continue; }
wget -q -P out/packages "$download_url" || {
echo "::error::Download failed: $filename"
exit 1
}
GOT_DEB=1
pool=$(deb_pool_for "$comp")
mkdir -p "$pool"
Expand Down Expand Up @@ -296,7 +277,10 @@ for asset in data.get('assets', []):
*) continue ;;
esac
if [ "$downloaded" -eq 0 ]; then
wget -q -P out/packages "$download_url" || { echo "::warning::download failed: $filename"; break; }
wget -q -P out/packages "$download_url" || {
echo "::error::Download failed: $filename"
exit 1
}
downloaded=1
GOT_RPM=1
fi
Expand All @@ -309,7 +293,12 @@ for asset in data.get('assets', []):
*)
# Non-package assets (SHA256SUMS, manifest.txt, ...) are mirrored only
# for the primary release, which is what the site links to.
[ "$tag" = "$PRIMARY_TAG" ] && wget -q -P out/packages "$download_url" ;;
if [ "$tag" = "$PRIMARY_TAG" ]; then
wget -q -P out/packages "$download_url" || {
echo "::error::Download failed: $filename"
exit 1
}
fi ;;
esac
done < "$ASSETS_FILE"

Expand Down Expand Up @@ -353,7 +342,7 @@ for pool in "$RPM_POOL_RHEL8" "$RPM_POOL_RHEL9"; do
done

if [ "$GOT_DEB" = "1" ]; then
echo "Building APT repository with multiple distribution components..."
echo "Building APT repository with active and retired compatibility components..."
pushd out/deb >/dev/null

if [ -d "pool/ubuntu22" ] && [ "$(ls -A pool/ubuntu22/*.deb 2>/dev/null)" ]; then
Expand Down Expand Up @@ -472,35 +461,35 @@ if [ "$GOT_DEB" = "1" ]; then
dpkg-scanpackages --arch arm64 pool/ubuntu24/ > "${DEB_DISTS_UBUNTU24_ARM64}/Packages"
gzip -k -f "${DEB_DISTS_UBUNTU24_ARM64}/Packages"
fi

# Keep retired component URLs valid without retaining packages from older
# releases. Existing apt sources can refresh cleanly, but package lookup
# returns no stale binaries.
for component in $APT_METADATA_COMPONENTS; do
for arch in amd64 arm64; do
metadata_dir="${DEB_DISTS}/${component}/binary-${arch}"
packages_file="${metadata_dir}/Packages"
mkdir -p "$metadata_dir"
if [ ! -f "$packages_file" ]; then
: > "$packages_file"
gzip -k -f "$packages_file"
fi
done
done

pushd "${DEB_DISTS}" >/dev/null

echo "Creating Release file"
# Determine which components we actually have
AVAILABLE_COMPONENTS=""
[ -d "${COMPONENTS}/binary-amd64" ] && AVAILABLE_COMPONENTS="${AVAILABLE_COMPONENTS} ${COMPONENTS}"
[ -d "deb11/binary-amd64" ] && AVAILABLE_COMPONENTS="${AVAILABLE_COMPONENTS} deb11"
[ -d "deb12/binary-amd64" ] && AVAILABLE_COMPONENTS="${AVAILABLE_COMPONENTS} deb12"
[ -d "deb13/binary-amd64" ] && AVAILABLE_COMPONENTS="${AVAILABLE_COMPONENTS} deb13"
[ -d "ubuntu22/binary-amd64" ] && AVAILABLE_COMPONENTS="${AVAILABLE_COMPONENTS} ubuntu22"
[ -d "ubuntu24/binary-amd64" ] && AVAILABLE_COMPONENTS="${AVAILABLE_COMPONENTS} ubuntu24"
AVAILABLE_COMPONENTS=$(echo $AVAILABLE_COMPONENTS | sed 's/^ *//')

# Determine available architectures
AVAILABLE_ARCHITECTURES=""
[ -d "${COMPONENTS}/binary-amd64" ] || [ -d "deb11/binary-amd64" ] || [ -d "deb12/binary-amd64" ] || [ -d "deb13/binary-amd64" ] || [ -d "ubuntu22/binary-amd64" ] || [ -d "ubuntu24/binary-amd64" ] && AVAILABLE_ARCHITECTURES="${AVAILABLE_ARCHITECTURES} amd64"
[ -d "${COMPONENTS}/binary-arm64" ] || [ -d "deb11/binary-arm64" ] || [ -d "deb12/binary-arm64" ] || [ -d "deb13/binary-arm64" ] || [ -d "ubuntu22/binary-arm64" ] || [ -d "ubuntu24/binary-arm64" ] && AVAILABLE_ARCHITECTURES="${AVAILABLE_ARCHITECTURES} arm64"
AVAILABLE_ARCHITECTURES=$(echo $AVAILABLE_ARCHITECTURES | sed 's/^ *//')

{
echo "Origin: ${ORIGIN}"
echo "Label: DocumentDB"
echo "Suite: ${SUITE}"
echo "Codename: ${SUITE}"
echo "Version: 1.0"
echo "Architectures: ${AVAILABLE_ARCHITECTURES}"
echo "Components: ${AVAILABLE_COMPONENTS}"
echo "Description: ${DESCRIPTION} - Multiple distributions supported"
echo "Architectures: amd64 arm64"
echo "Components: ${APT_METADATA_COMPONENTS}"
echo "Description: ${DESCRIPTION}"
echo "Date: $(date -Ru)"
generate_hashes MD5Sum md5sum
generate_hashes SHA1 sha1sum
Expand All @@ -526,7 +515,7 @@ if [ "$GOT_DEB" = "1" ]; then
popd >/dev/null


echo "APT repository built successfully with multiple distribution support"
echo "APT repository built successfully"
fi

if [ "$GOT_RPM" = "1" ]; then
Expand All @@ -544,52 +533,48 @@ if [ "$GOT_RPM" = "1" ]; then
fi

for POOL in "$RHEL8_POOL" "$RHEL9_POOL"; do
if [ -d "$POOL" ] && [ "$(find "$POOL" -name "*.rpm" -type f | wc -l)" -gt 0 ]; then
echo "Processing YUM repository: $POOL"
pushd "$POOL" >/dev/null

if [ -n "$GPG_FINGERPRINT" ]; then
for rpm_file in *.rpm; do
rpm --define "%_signature gpg" --define "%_gpg_name ${GPG_FINGERPRINT}" --addsign "$rpm_file" 2>/dev/null || true
done
fi

echo "Running createrepo_c in $(pwd)"
if createrepo_c .; then
echo "Repository metadata created successfully"
ls -la repodata/ 2>/dev/null || echo "No repodata directory found"
else
echo "ERROR: createrepo_c failed for $POOL"
fi

if [ -n "$GPG_FINGERPRINT" ] && [ -f repodata/repomd.xml ]; then
gpg --default-key "$GPG_FINGERPRINT" --detach-sign --armor repodata/repomd.xml 2>/dev/null || true
fi

popd >/dev/null
else
echo "Skipping $POOL: directory not found or no RPM files"
mkdir -p "$POOL"
echo "Processing YUM repository: $POOL"
pushd "$POOL" >/dev/null

if [ -n "$GPG_FINGERPRINT" ]; then
for rpm_file in *.rpm; do
[ -f "$rpm_file" ] || continue
rpm --define "%_signature gpg" --define "%_gpg_name ${GPG_FINGERPRINT}" --addsign "$rpm_file"
signature=$(rpm -qp --qf '%{RSAHEADER:pgpsig}' "$rpm_file" 2>/dev/null)
if [ -z "$signature" ] || [ "$signature" = "(none)" ]; then
echo "::error::$rpm_file was not signed"
exit 1
fi
done
fi

echo "Running createrepo_c in $(pwd)"
createrepo_c .

if [ -n "$GPG_FINGERPRINT" ]; then
gpg --batch --yes --default-key "$GPG_FINGERPRINT" \
--detach-sign --armor repodata/repomd.xml
fi

popd >/dev/null
done

# Create main repository for backward compatibility
if [ -d "$RHEL8_POOL" ] && [ "$(find "$RHEL8_POOL" -name "*.rpm" -type f | wc -l)" -gt 0 ]; then
echo "Creating main YUM repository"
mkdir -p "$MAIN_POOL"
# Keep the legacy main URL valid. It mirrors RHEL 8 only when the selected
# release actually contains RHEL 8 packages; otherwise it is an empty
# compatibility repository.
echo "Creating main YUM compatibility repository"
mkdir -p "$MAIN_POOL"
if [ "$(find "$RHEL8_POOL" -name "*.rpm" -type f | wc -l)" -gt 0 ]; then
cp "$RHEL8_POOL"/*.rpm "$MAIN_POOL"/ 2>/dev/null || true
pushd "$MAIN_POOL" >/dev/null
echo "Running createrepo_c for main repository in $(pwd)"
if createrepo_c .; then
echo "Main repository metadata created successfully"
ls -la repodata/ 2>/dev/null || echo "No repodata directory found"
else
echo "ERROR: createrepo_c failed for main repository"
fi
if [ -n "$GPG_FINGERPRINT" ] && [ -f repodata/repomd.xml ]; then
gpg --default-key "$GPG_FINGERPRINT" --detach-sign --armor repodata/repomd.xml 2>/dev/null || true
fi
popd >/dev/null
fi
pushd "$MAIN_POOL" >/dev/null
createrepo_c .
if [ -n "$GPG_FINGERPRINT" ]; then
gpg --batch --yes --default-key "$GPG_FINGERPRINT" \
--detach-sign --armor repodata/repomd.xml
fi
popd >/dev/null

echo "YUM repositories built successfully"
fi
Expand All @@ -598,6 +583,7 @@ fi
echo "Package repository setup complete!"
echo ""
echo "Repository URLs:"
echo " APT: https://documentdb.io/deb stable main"
echo " YUM: https://documentdb.io/rpm/rhel8 (or /rhel9, /main)"
echo " APT: https://documentdb.io/deb stable ubuntu24"
echo " YUM: https://documentdb.io/rpm/rhel9"
echo " Retired URLs retain empty metadata for package-manager compatibility."
echo " Browse: https://documentdb.io/packages/"
Loading