Skip to content

Commit

Permalink
ENH: Transition SourceTarball.bash script for .cid
Browse files Browse the repository at this point in the history
We fetch from a local IPFS gateway to ensure the data can be obtained
via IPFS. This also ensures the script will run offline, and it runs
much faster. The IPFS gateway will verify CIDs; we do not need to verify
them separately.

Re: #3760
  • Loading branch information
thewtex committed Nov 18, 2023
1 parent 0fcffbc commit 561383e
Show file tree
Hide file tree
Showing 2 changed files with 8 additions and 11 deletions.
5 changes: 5 additions & 0 deletions Documentation/Maintenance/Release.md
Expand Up @@ -358,6 +358,9 @@ endings.

The `InsightData` tarballs are generated along with the source code tarballs.

Data is fetched from [IPFS]. An IPFS daemon must be running to fetch the data
- [ipfs-desktop] is recommended.

Once the repository has been tagged, we use the following script in the
repository to create the tarballs:

Expand Down Expand Up @@ -873,6 +876,8 @@ excellent packaging.
[documentation page]: https://www.itk.org/ITK/help/documentation.html
[download page]: https://itk.org/ITK/resources/software.html
[GitHub]: https://github.com/InsightSoftwareConsortium/ITK
[IPFS]: https://ipfs.tech/
[ipfs-desktop]: https://github.com/ipfs/ipfs-desktop/releases
[ITKPythonPackage]: https://itkpythonpackage.readthedocs.io/en/latest/index.html
[ITK discussion]: https://discourse.itk.org/
[Image.sc Forum]: https://image.sc
Expand Down
14 changes: 3 additions & 11 deletions Utilities/Maintenance/SourceTarball.bash
Expand Up @@ -36,27 +36,20 @@ return_pipe_status() {

find_data_objects() {
git ls-tree --full-tree -r "$1" |
egrep '\.(md5)$' |
egrep '\.(cid)$' |
while read mode type obj path; do
case "$path" in
*.md5) echo MD5/$(git cat-file blob $obj) ;;
*.cid) echo CID/$(git cat-file blob $obj) ;;
*) die "Unknown ExternalData content link: $path" ;;
esac
done | sort | uniq
return_pipe_status
}

validate_MD5() {
md5sum=$(md5sum "$1" | sed 's/ .*//') &&
if test "$md5sum" != "$2"; then
die "Object MD5/$2 is corrupt: $1"
fi
}

download_object() {
algo="$1" ; hash="$2" ; path="$3"
mkdir -p $(dirname "$path") &&
if curl -L "https://www.itk.org/files/ExternalData/$algo/$hash" -o "$path.tmp$$" 1>&2; then
if curl -L "http://127.0.01:8080/ipfs/$hash" -o "$path.tmp$$" 1>&2; then
mv "$path.tmp$$" "$path"
else
rm -f "$path.tmp$$"
Expand All @@ -78,7 +71,6 @@ index_data_objects() {
download_object "$algo" "$hash" "$path" &&
file="$path"
fi &&
validate_$algo "$file" "$hash" &&
obj=$(git hash-object -t blob -w "$file") &&
echo "100644 blob $obj $path" ||
return
Expand Down

0 comments on commit 561383e

Please sign in to comment.