Skip to content

Commit

Permalink
Wallentx/bb harvest integration (#368)
Browse files Browse the repository at this point in the history
* Cleanup commits/rebase

* Testing CI

* CI

* CI

* CI

* CI

* CI

* CI

* CI

* CI

* CI

* CI

* CI

* CI

* CI

* CI

* CI

* CI

* CI

* CI

* CI

* CI

* CI

* CI

* CI

* CI

* CI

* CI

* CI

* CI

* Changing to manylinux2014

* Update bladebit harvester sha256

* Remove bladebit building ref

* Fixing more windows stuff

* Change trigger CI branch

* Some more build fixes for Windows

* More cmake fixes

* Updating CI for windows

* CI

* Fix packaging location of bladebit_harcester on linux

* Reabling macos

* Re-disabling macos

* Remove this branch CI trigger

* Re-enabling pre-release logic CI (#361)

* Re-enabling pre-release logic CI

* Lint fixes

* Lint fixes2

---------

Co-authored-by: Harold Brenes <h.brenes@chia.net>

* Fix missing decompressor queue when no compression

* Update Catch2 to v3.3.2. (#346)

* correct wheel matrix arch -> arm (#365)

* Updating the ways bladebit harvester gets included

* Remove packaging of bladebit harvester dlls into wheel.
Opting for static lib instead.

* Remove shutil

* More logging on GRResult_OK not ok. (#366)

* Copy .a instead of .so GreenReaper artifact

* Fix no decompressor deallocation when
it failues to pre-allocate decompression buffers.

* Fix linking and building of bladebit harvester

* Add action to fetch bladebit harvester

* Fixing fetch_bladebit_harvester.sh on linux

* Enabling all builds

* Fixing macos multiline

* Correct build opts for GR in win build extension

* Re-add bladebit_harvester.dll copy into wheel code

---------

Co-authored-by: Harold Brenes <h.brenes@chia.net>
Co-authored-by: Amine Khaldi <amine.khaldi@reactos.org>
Co-authored-by: Kyle Altendorf <sda@fstab.net>
Co-authored-by: Chris Marslender <chrismarslender@gmail.com>
Co-authored-by: Florin Chirica <fchirica96@gmail.com>
  • Loading branch information
6 people committed Jun 29, 2023
1 parent 51d6a32 commit 742d5fa
Show file tree
Hide file tree
Showing 6 changed files with 548 additions and 197 deletions.
79 changes: 79 additions & 0 deletions .github/actions/fetch_bladebit_harvester.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
#!/usr/bin/env bash
set -eo pipefail
_dir=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd)
cd "$_dir/../.."

##
# Usage: fetch_bladebit_harvester.sh <linux|macos|windows> <arm64|x86-64>
#
# Use gitbash or similar under Windows.
##
host_os=$1
host_arch=$2

if [[ "${host_os}" != "linux" ]] && [[ "${host_os}" != "macos" ]] && [[ "${host_os}" != "windows" ]]; then
echo >&2 "Unkonwn OS '${host_os}'"
exit 1
fi

if [[ "${host_arch}" != "arm64" ]] && [[ "${host_arch}" != "x86-64" ]]; then
echo >&2 "Unkonwn Architecture '${host_arch}'"
exit 1
fi

# Update these when pointing to different releases
artifact_ver="v3.0.0-alpha4"
artifact_base_url="https://github.com/harold-b/bladebit-test/releases/download/v3-alpha4-fixes"

linux_sha256=
macos_sha256=
windows_sha256=

artifact_ext="tar.gz"
sha_bin="sha256sum"
expected_sha256=

case "${host_os}" in
linux)
expected_sha256=$linux_sha256
;;
macos)
expected_sha256=$macos_sha256
sha_bin="shasum -a 256"
;;
windows)
expected_sha256=$windows_sha256
artifact_ext="zip"
;;
*)
echo >&2 "Unexpected OS '${host_os}'"
exit 1
;;
esac

# Download artifact
artifact_name="green_reaper.${artifact_ext}"
curl -L "${artifact_base_url}/green_reaper-${artifact_ver}-${host_os}-${host_arch}.${artifact_ext}" >"${artifact_name}"

# Validate sha256, if one was given
if [ -n "${expected_sha256}" ]; then
gr_sha256="$(${sha_bin} ${artifact_name})"

if [[ "${gr_sha256}" != "${expected_sha256}" ]]; then
echo >&2 "GreenReaper SHA256 mismatch!"
echo >&2 " Got : '${gr_sha256}'"
echo >&2 " Expected: '${expected_sha256}'"
exit 1
fi
fi

# Unpack artifact
dst_dir="libs/green_reaper"
mkdir -p "${dst_dir}"
if [[ "${artifact_ext}" == "zip" ]]; then
unzip -d "${dst_dir}" "${artifact_name}"
else
pushd "${dst_dir}"
tar -xzvf "../../${artifact_name}"
popd
fi
87 changes: 64 additions & 23 deletions .github/workflows/build-wheels.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@ on:
push:
branches:
- main
tags:
- '**'
release:
types: [published]
pull_request:
branches:
- '**'
Expand All @@ -19,6 +19,7 @@ jobs:
build-wheels:
name: Wheel - ${{ matrix.os.name }} ${{ matrix.python.major-dot-minor }} ${{ matrix.arch.name }}
runs-on: ${{ matrix.os.runs-on[matrix.arch.matrix] }}
continue-on-error: true
strategy:
fail-fast: false
matrix:
Expand All @@ -44,31 +45,31 @@ jobs:
- major-dot-minor: '3.7'
cibw-build: 'cp37-*'
manylinux:
arch: manylinux2014
intel: manylinux2010
arm: manylinux2014
intel: manylinux2014
matrix: '3.7'
- major-dot-minor: '3.8'
cibw-build: 'cp38-*'
manylinux:
arch: manylinux2014
intel: manylinux2010
arm: manylinux2014
intel: manylinux2014
matrix: '3.8'
- major-dot-minor: '3.9'
cibw-build: 'cp39-*'
manylinux:
arch: manylinux2014
intel: manylinux2010
arm: manylinux2014
intel: manylinux2014
matrix: '3.9'
- major-dot-minor: '3.10'
cibw-build: 'cp310-*'
manylinux:
arch: manylinux2014
intel: manylinux2010
arm: manylinux2014
intel: manylinux2014
matrix: '3.10'
- major-dot-minor: '3.11'
cibw-build: 'cp311-*'
manylinux:
arch: manylinux2014
arm: manylinux2014
intel: manylinux2014
matrix: '3.11'
arch:
Expand Down Expand Up @@ -112,6 +113,12 @@ jobs:
with:
fetch-depth: 0

- name: Set Env
if: env.RUNNER_ARCH != 'ARM64'
uses: Chia-Network/actions/setjobenv@main
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

- uses: Chia-Network/actions/setup-python@main
with:
python-version: ${{ matrix.python.major-dot-minor }}
Expand All @@ -120,6 +127,16 @@ jobs:
run: |
pip install pipx
- name: Get Windows Bladebit Harvester Artifact
if: runner.os == 'Windows'
shell: bash
run: |
set -eo pipefail
set -x
windows_sha256='8ad2710faad4a41f72de20707539f024f02db079968ea546699f4ded56fe9a57'
.github/actions/fetch_bladebit_harvester.sh windows x86-64
- name: Build and test
env:
CIBW_PRERELEASE_PYTHONS: True
Expand All @@ -130,22 +147,41 @@ jobs:
CIBW_SKIP: '*-manylinux_i686 *-win32 *-musllinux_*'
CIBW_MANYLINUX_AARCH64_IMAGE: ${{ matrix.python.manylinux['arm'] }}
CIBW_MANYLINUX_X86_64_IMAGE: ${{ matrix.python.manylinux['intel'] }}
CIBW_ENVIRONMENT_LINUX: "PATH=/project/cmake-3.17.3-Linux-`uname -m`/bin:$PATH"
CIBW_BEFORE_ALL_LINUX: >
curl -L https://github.com/Kitware/CMake/releases/download/v3.17.3/cmake-3.17.3-Linux-`uname -m`.sh > cmake.sh
&& yes | sh cmake.sh | cat
&& rm -f /usr/bin/cmake
&& which cmake
&& cmake --version
&& uname -a
CIBW_ENVIRONMENT_WINDOWS: "CP_USE_GREEN_REAPER=1"
CIBW_ENVIRONMENT_LINUX: CP_USE_GREEN_REAPER="1"
CIBW_BEFORE_ALL_LINUX: |
set -eo pipefail
set -x
# Get bladebit harvester
set -eo pipefail
ARCH=$(uname -m)
if [[ $ARCH == x86_64 ]]; then
linux_sha256='3d0cc6e62a7936966789349c4bc573882866a3f1a9f5e365e4961e7bea80831b'
.github/actions/fetch_bladebit_harvester.sh linux x86-64
else
linux_sha256='637c6b073c26d87be513d1e04a97f4e6cc6b27532dc716cc462ee7fbaae31470'
.github/actions/fetch_bladebit_harvester.sh linux arm64
fi
CIBW_BEFORE_BUILD_LINUX: >
python -m pip install --upgrade pip
CIBW_ARCHS_MACOS: ${{ matrix.os.cibw-archs-macos[matrix.arch.matrix] }}
CIBW_BEFORE_ALL_MACOS: >
CIBW_BEFORE_ALL_MACOS: |
brew install gmp boost cmake
# Get bladebit harvester
set -eo pipefail
ARCH=$(uname -m)
if [[ $ARCH == x86_64 ]]; then
macos_sha256='a76d7da43c6e045f3d0a57a51924211283ba60880558dbbb636d9601c62f6390'
.github/actions/fetch_bladebit_harvester.sh macos x86-64
else
macos_sha256='6894b1b3ebae13a3233b9bc31d2b2a179078ba67595b8be0ee9a69a2a75bec23'
.github/actions/fetch_bladebit_harvester.sh macos arm64
fi
CIBW_BEFORE_BUILD_MACOS: >
python -m pip install --upgrade pip
CIBW_ENVIRONMENT_MACOS: "MACOSX_DEPLOYMENT_TARGET=10.14"
CIBW_ENVIRONMENT_MACOS: "MACOSX_DEPLOYMENT_TARGET=10.14 CP_USE_GREEN_REAPER=1"
CIBW_TEST_REQUIRES: pytest
CIBW_TEST_COMMAND: py.test -v {project}/tests
run:
Expand Down Expand Up @@ -274,6 +310,11 @@ jobs:
with:
fetch-depth: 0

- name: Set Env
uses: Chia-Network/actions/setjobenv@main
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

- uses: Chia-Network/actions/setup-python@main
with:
python-version: ${{ matrix.python.major-dot-minor }}
Expand All @@ -298,15 +339,15 @@ jobs:
run: pip install twine

- name: Publish distribution to PyPI
if: startsWith(github.event.ref, 'refs/tags') && steps.check_secrets.outputs.HAS_SECRET
if: env.RELEASE == 'true' && steps.check_secrets.outputs.HAS_SECRET
env:
TWINE_USERNAME: __token__
TWINE_NON_INTERACTIVE: 1
TWINE_PASSWORD: ${{ secrets.pypi_password }}
run: twine upload --non-interactive --skip-existing --verbose 'dist/*'

- name: Publish distribution to Test PyPI
if: steps.check_secrets.outputs.HAS_SECRET
if: env.PRE_RELEASE == 'true' && steps.check_secrets.outputs.HAS_SECRET
env:
TWINE_REPOSITORY_URL: https://test.pypi.org/legacy/
TWINE_USERNAME: __token__
Expand Down
5 changes: 4 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -27,4 +27,7 @@ build
venv
build-tsan
build-*
cmake-build*
cmake-build*
*.zip
*.tar.gz
libs/
Loading

0 comments on commit 742d5fa

Please sign in to comment.