Skip to content

Commit

Permalink
Merge pull request #550 from norlab-ulaval/dev-feat-add-dockerhub-rel…
Browse files Browse the repository at this point in the history 
…ease-logic-NMO-469

Add dockerhub release logic and improve main readme
  • Loading branch information
@RedLeader962
RedLeader962 committed Jan 18, 2024
2 parents a94cb28 + 6de0d75 commit 65320bc
Show file tree
Hide file tree
Showing 29 changed files with 441 additions and 173 deletions.
23 changes: 4 additions & 19 deletions .github/pull_request_template.md
View file
Expand Up @@ -2,6 +2,7 @@

### Summary:


### Changes and type of changes (quick overview):

-
Expand All @@ -19,8 +20,7 @@
- [ ] I have commented hard-to-understand code
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] All tests pass locally with my changes
(Check [README.md #Contributing](https://github.com/norlab-ulaval/libpointmatcher/tree/develop#contributing)
for local testing procedure using _libpointmatcher-build-system_)
(Check [contributing_instructions.md](/doc/contributing/contributing_instructions.md) for local testing procedure using _libpointmatcher-build-system_)

### PR creation related

Expand All @@ -33,22 +33,7 @@

- [ ] I have included a quick summary of the changes
- [ ] I have indicated the related issue's id with `# <issue-id>` if changes are of type `fix`
- [ ] I have included a high-level list of changes and their corresponding type
- Types: `feat` `fix` `docs` `style` `refactor` `perf` `test` `build` `ci` `chore` `revert`
- Breaking changes: `<type>!`
- Reference:
see [commit_msg_reference.md](https://github.com/norlab-ulaval/libpointmatcher/blob/develop/commit_msg_reference.md)
in the repository root for details
- [ ] I have included a high-level list of changes and their corresponding types
(See [commit_msg_reference.md](/doc/contributing/commit_msg_reference.md) for details)

---

## Note for repository admins

### Release PR related

- Only repository admins have the privilege to `push/merge` on the default branch (ie: `master`)
and the `release` branch.
- Keep PR in `draft` mode until all the release reviewers are ready to push the release.
- Once a PR from `release` -> `master` branch is created (not in draft mode),
- it triggers the _build-system_ test
- (in-progress) and it triggers the _semantic release automation_
5 changes: 2 additions & 3 deletions .github/workflows/build-python.yaml
View file
Expand Up @@ -177,7 +177,7 @@ jobs:
if: ${{ steps.cache-boost.outputs.cache-hit != 'true' && runner.os == 'Linux' }}
working-directory: ${{ env.BOOST_SRC_DIR }}
run: |
wget --no-verbose https://boostorg.jfrog.io/artifactory/main/release/${{ env.BOOST_VERSION }}/source/${{ env.BOOST_ARCHIVE_NAME }}
wget --no-verbose https://archives.boost.io/release/${{ env.BOOST_VERSION }}/source/${{ env.BOOST_ARCHIVE_NAME }}
7z -y x ${{ env.BOOST_ARCHIVE_NAME }}
rm ${{ env.BOOST_ARCHIVE_NAME }}
Expand All @@ -192,7 +192,7 @@ jobs:
if: steps.cache-libnabo.outputs.cache-hit != 'true'
working-directory: ${{ env.LIBNABO_SRC_DIR }}
run: |
git clone -b ${{ env.LIBNABO_VERSION }} --single-branch https://github.com/ethz-asl/libnabo.git
git clone -b ${{ env.LIBNABO_VERSION }} --single-branch https://github.com/norlab-ulaval/libnabo.git
- name: Install libnabo ${{ env.LIBNABO_VERSION }} on Windows
if: ${{ steps.cache-libnabo.outputs.cache-hit != 'true' && runner.os == 'Windows' }}
Expand Down Expand Up @@ -319,4 +319,3 @@ jobs:
prerelease: true
allowUpdates: true
generateReleaseNotes: true

178 changes: 66 additions & 112 deletions README.md
View file
Expand Up @@ -2,6 +2,26 @@

![banner](doc/images/banner_light.jpeg)

[//]: # ( ==== Description ====================================================================== )
**_libpointmatcher_ is a modular library implementing the Iterative Closest Point (ICP) algorithm<br>
for aligning point clouds. It has applications in robotics and computer vision.**
<br>
The library is written in C++ for efficiency with [bindings in Python](doc/index.md#python-).
<br>
<br>

[//]: # (====Badges===============================================================================)

<img alt="GitHub Repo stars" src="https://img.shields.io/github/stars/norlab-ulaval/libpointmatcher">
<img alt="GitHub forks" src="https://img.shields.io/github/forks/norlab-ulaval/libpointmatcher">
<img alt="GitHub License" src="https://img.shields.io/github/license/norlab-ulaval/libpointmatcher">
<img alt="GitHub release (with filter)" src="https://img.shields.io/github/v/release/norlab-ulaval/libpointmatcher">
<a href="http://132.203.26.125:8111"><img src="https://img.shields.io/static/v1?label=JetBrains TeamCity&message=CI/CD&color=green?style=plastic&logo=teamcity" /></a>
<a href="https://hub.docker.com/repository/docker/norlabulaval/libpointmatcher/"> <img alt="Docker Image Version (latest semver)" src="https://img.shields.io/docker/v/norlabulaval/libpointmatcher?logo=docker&label=libpointmatcher"> </a>

<br>
<br>

[//]: # (====Awesome badges=======================================================================)

[![Mentioned in Awesome LIDAR](https://awesome.re/mentioned-badge.svg)](https://github.com/szenergy/awesome-lidar#basic-matching-algorithms)
Expand All @@ -18,36 +38,29 @@
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
<a href="http://jslee02.github.io/awesome-robotics-libraries/#3d-mapping">Robotics Libraries</a>
</sup>


[//]: # (====GitHub badges========================================================================)

<img alt="GitHub Repo stars" src="https://img.shields.io/github/stars/norlab-ulaval/libpointmatcher">
<img alt="GitHub forks" src="https://img.shields.io/github/forks/norlab-ulaval/libpointmatcher">
<img alt="GitHub License" src="https://img.shields.io/github/license/norlab-ulaval/libpointmatcher">
<img alt="GitHub release (with filter)" src="https://img.shields.io/github/v/release/norlab-ulaval/libpointmatcher">
<img src="https://img.shields.io/static/v1?label=JetBrains TeamCity&message=CI/CD&color=green?style=plastic&logo=teamcity" />
<br>
<br>
<hr style="color:lightgray;background-color:lightgray">
</div>


[//]: # (====Supported OS and aarch===============================================================)

`libpointmatcher` is tested on our build system under the following architecture and OS:
- x86 and arm64/v8
### Supported OS And Architecture
_libpointmatcher_ is tested on our build system under the following architecture and OS:
- Ubuntu bionic (18.04) and focal (20.04)
- x86 and arm64/v8

Note:
- support for Ubuntu jammy (22.04) comming soon
- `libpointmatcher` reportedly works on MacOs OsX (latest) and Windows (latest)
- _libpointmatcher_ reportedly works on MacOs OsX (latest) and Windows (latest)


---

[//]: # (====Release note=========================================================================)

### ★ Version `1.4.0` release note (important)
### ★ Version `1.4.0` Release Note
This release of _libpointmatcher_ introduces the integration of [norlab-build-system (NBS)](https://github.com/norlab-ulaval/norlab-build-system) as a _git submodule_ for codebase development and testing.

Execute the following to clone the repository with its submodule:
Expand All @@ -59,11 +72,14 @@ If _libpointmatcher_ was previously cloned, execute the following to fetch its n
git submodule update --remote --recursive --init
```

### ★ Contributing Instructions
See [contributing_instructions.md](doc/contributing/contributing_instructions.md)
for instructions related to bug reporting, code contribution and for setting up the `libpointmatcher-build-system`
on your workstation to speed up your local development workflow.


[//]: # (====Body=================================================================================)
# Documentation and Tutorials

libpointmatcher is a modular library implementing the Iterative Closest Point (ICP) algorithm for aligning point clouds. It has applications in robotics and computer vision.
The library is written in C++ for effeciency with [bindings in Python](https://github.com/norlab-ulaval/libpointmatcher/blob/master/doc/index.md#python-).

**Quick link for the tutorial pages: [Tutorials](doc/index.md)** (also available on [readthedocs.org](http://libpointmatcher.readthedocs.org/) but might not be up-to-date).

Expand Down Expand Up @@ -93,9 +109,41 @@ and was compiled on:
* Mac OS X ([see how](/doc/CompilationMac.md))
* Windows ([see how](/doc/CompilationWindows.md) - partially supported)

### Docker images

Run the following commands to pull and run libpointmatcher in a docker container

```shell
docker pull norlabulaval/libpointmatcher:latest-ubuntu-focal

docker run -it --rm norlabulaval/libpointmatcher:latest-ubuntu-focal
```

See
available [libpointmatcher image tags](https://hub.docker.com/repository/docker/norlabulaval/libpointmatcher/)
on dockerhub.

To install docker related dependencies on ubuntu, execute the following
```shell
cd ./build_system/lpm_utility_script

# Execute docker tools install script i.e. docker daemon, docker compose, docker buildx
bash lpm_install_docker_tools.bash
```


### Compilation & Installation

For beginner users unfamiliar with compiling and installing a library in Linux, go [here](doc/CompilationUbuntu.md) for detailed instructions on compiling libpointmatcher from the source code.

For conveniences, you can use the provided installer script for ubuntu
```shell
bash libpointmatcher_dependencies_installer.bash

# Use the --help flag to see the list of optional flag
bash libpointmatcher_installer.bash [<optional flag>]
```

If you are comfortable with Linux and CMake and have already installed the prerequisites above, the following commands should install libpointmatcher on your system.

```bash
Expand All @@ -105,6 +153,7 @@ make
sudo make install
```


### Testing

Libpointmatcher ships with a version of the Google testing framework [GTest](https://github.com/google/googletest). Unit tests are located in the `utest/` directory and are compiled with libpointmatcher (CMake variable `BUILD_TESTS` must be set to `TRUE` before compiling). To run the tests and make sure that your compiled version is working correctly, run the test executable in your build directory:
Expand Down Expand Up @@ -133,101 +182,6 @@ tutorial [Importing and Exporting Point Clouds](doc/ImportExport.md). Example ex
those file formats from the command line can be found in the `/examples` directory and are
described [here](doc/ICPIntro.md) in more detail.

---

# Contributing

## Bug reporting

Please use our [github's issue tracker](http://github.com/ethz-asl/libpointmatcher/issues) to report bugs. If you are running the library on Ubuntu, copy-paste the output of the script [listVersionsUbuntu.sh](https://github.com/norlab-ulaval/libpointmatcher/blob/master/utest/listVersionsUbuntu.sh) to simplify the search of an answer.

## Codebase development

Libpointmatcher codebase now integrate [norlab-build-system (NBS)](https://github.com/norlab-ulaval/norlab-build-system) and [norlab-shell-script-tools (N2ST)](https://github.com/norlab-ulaval/norlab-shell-script-tools).
`NBS` is a build-infrastructure-agnostic build system custom-made to meet our needs in robotic software engineering at NorLab and `N2ST` is a library of shell script functions as well as a shell testing tools leveraging _**bats-core**_ and _**docker**_ .
`N2ST` purpose is to speed up shell script development and improve reliability.

`NBS` is deployed on our [TeamCity](https://www.jetbrains.com/teamcity/) continuous integration/deployment server and oversees protected branches of the [libpointmatcher](https://github.com/norlab-ulaval/libpointmatcher) GitHub repository:

- The `develop` branch can only be merged through a pull-request from any `<feature>` branches. Any contributor can submit a pull request to the `develop` branch;
- the `release` branch is a revision and preparation branch where we can freeze the codebase in a given state without stalling to `develop` branch progression;
- The `master` branch can only be merged through a pull-request from the `release` branch. Only repository admin can submit a PR to the `master` branch.

In any cases, submitting a pull request to `develop` or `master` will trigger a build/test configuration on our build system and the pull request will be granted if the build/test run succeed.

**Current build matrix:**
`[latest] x [x86, arm64] x [ubuntu] x [bionic, focal] x [Release, RelWithDebInfo, MinSizeRel]`

### Development workflow

To speed up the development process, you can run the build system localy on your workstation and have access to stacktrace and build log.
It support multi-OS and multi-architecture through docker container.

#### Install _libpointmatcher-build-system_ dependencies
```shell
cd <path/to/libpointmatcher>

# If libpointmatcher is already cloned, fetch the NBS and N2ST submodule
git submodule update --remote --recursive --init

cd ./build_system/lpm_utility_script

# Execute docker tools install script i.e. docker daemon, docker compose, docker buildx
bash lpm_install_docker_tools.bash

# Configure a multi-architecture docker builder
bash lpm_create_multiarch_docker_builder.bash
```

#### libpointmatcher development › to execute build/test step locally
```shell
cd <path/to/libpointmatcher>/build_system

# Run the build matrix as specified in ".env.build_matrix.libpointmatcher"
# on native architecture using "ci_PR" service
bash lpm_crawl_libpointmatcher_build_matrix.bash --fail-fast -- build ci_PR

# Run a specific case using build flags with multi-architecture
# virtualization using "ci_PR_amd64" and "ci_PR_arm64v8" services
bash lpm_crawl_libpointmatcher_build_matrix.bash \
--repository-version-build-matrix-override latest \
--os-name-build-matrix-override ubuntu \
--cmake-build-type-build-matrix-override RelWithDebInfo \
--ubuntu-version-build-matrix-override focal \
--fail-fast \
-- build ci_PR_amd64 ci_PR_arm64v8

# Read the help for details
bash lpm_crawl_libpointmatcher_build_matrix.bash --help
```
Note: To assess the state of the codebase, even for cases that are known the break the build,
execute `lpm_crawl_libpointmatcher_build_matrix.bleeding.bash` with build matrix `.env.build_matrix.libpointmatcher.bleeding`.
The stable build matrix used for release is `.env.build_matrix.libpointmatcher`.


#### Build system development
```shell
cd <path/to/libpointmatcher>/build_system/tests/

# To execute docker dryrun and configuration tests
bash run_all_docker_dryrun_and_config_tests.bash

# To execute shell script tests
bash run_bats_core_test_in_n2st.bash

# To spin a container in interactive mode with the codebase cloned but not compiled
cd ./tests_docker_interactive/
bash build_and_run_IamBuildSystemTester.bash bash
```

#### Build system notes:
- `lpm_crawl_dependencies_build_matrix.bash` execute the build matrix for the libpointmatcher dependencies.
It's not required to build them locally as they are pre-build by our TeamCity server periodically push to dockerhub.
When executing `lpm_crawl_libpointmatcher_build_matrix.bash`, the `libpointmatcher-dependencies` docker images are pull and used as base image for the `libpointmatcher-[ci_PR_test|release]` images.
- About `libpointmatcher/.github/workflow/` vs `libpointmatcher/build_system/` logic: Those are separate build logic.
`.github/workflow/` was community contributed and as the responsibilities of building python-binding and pushing packages.
For this reason, it run a one-dimension build matrix: multiple python version, single OS version, single arch (x86) and
single compile flag which GitHub action computing resources can handle just fine.

---

Expand Down Expand Up @@ -301,7 +255,7 @@ libpointmatcher is released under a permissive BSD license. Enjoy!
[CMake documentation]: https://cmake.org/cmake/help/v3.10/
[git]: http://git-scm.com
[Eigen]: http://eigen.tuxfamily.org
[libnabo]: http://github.com/ethz-asl/libnabo
[libnabo]: https://github.com/norlab-ulaval/libnabo
[ROS]: http://www.ros.org/
[Paraview]: http://www.paraview.org/
[yaml-cpp]: https://github.com/jbeder/yaml-cpp
Expand Down
3 changes: 1 addition & 2 deletions build_system/.env.build_matrix.dependencies
View file
Expand Up @@ -11,8 +11,7 @@ NBS_EXECUTE_BUILD_MATRIX_OVER_COMPOSE_FILE=docker-compose.dependencies.yaml
# Libpointmatcher version
#
# 'latest' is the latest push to the libpointmatcher master branch
#NBS_MATRIX_REPOSITORY_VERSIONS=( 'v1.3.1' 'latest' 'v2.0.test' )
NBS_MATRIX_REPOSITORY_VERSIONS=( 'latest' )
NBS_MATRIX_REPOSITORY_VERSIONS=( '1.4.0' 'latest' )

#
# Libpointmatcher dependencies CMAKE_BUILD_TYPE
Expand Down
2 changes: 1 addition & 1 deletion build_system/.env.build_matrix.libpointmatcher
View file
Expand Up @@ -11,7 +11,7 @@ NBS_EXECUTE_BUILD_MATRIX_OVER_COMPOSE_FILE=docker-compose.libpointmatcher.yaml
# Libpointmatcher version
#
# 'latest' is the latest push to the libpointmatcher master branch
#NBS_MATRIX_REPOSITORY_VERSIONS=( 'v1.3.1' 'latest' 'v2.0.test' )
#NBS_MATRIX_REPOSITORY_VERSIONS=( '1.4.0' 'latest' )
NBS_MATRIX_REPOSITORY_VERSIONS=( 'latest' )

#
Expand Down
2 changes: 1 addition & 1 deletion build_system/.env.build_matrix.libpointmatcher.bleeding
View file
Expand Up @@ -11,7 +11,7 @@ NBS_EXECUTE_BUILD_MATRIX_OVER_COMPOSE_FILE=docker-compose.libpointmatcher.yaml
# Libpointmatcher version
#
# 'latest' is the latest push to the libpointmatcher master branch
#NBS_MATRIX_REPOSITORY_VERSIONS=( 'v1.3.1' 'latest' 'v2.0.test' )
#NBS_MATRIX_REPOSITORY_VERSIONS=( '1.4.0' 'latest' )
NBS_MATRIX_REPOSITORY_VERSIONS=( 'latest' )

#
Expand Down
38 changes: 38 additions & 0 deletions build_system/.env.build_matrix.libpointmatcher.release
View file
@@ -0,0 +1,38 @@
#
# Build matrix variables
#

#
# The compose file on which the build matrix will be crawled
#
NBS_EXECUTE_BUILD_MATRIX_OVER_COMPOSE_FILE=docker-compose.libpointmatcher.yaml

#
# Libpointmatcher version
#
# 'latest' is the latest push to the libpointmatcher master branch
NBS_MATRIX_REPOSITORY_VERSIONS=( '1.4.0' 'latest' )

#
# Libpointmatcher CMAKE_BUILD_TYPE
#
NBS_MATRIX_CMAKE_BUILD_TYPE=( 'Release' )

#
# LIBPOINTMATCHER supported OS
#
# ToDo: implement OsX support for arm64-Darwin (ref task NMO-213)
#NBS_MATRIX_SUPPORTED_OS=( 'ubuntu' 'l4t' 'osx' )
NBS_MATRIX_SUPPORTED_OS=( 'ubuntu' )

#
# ubuntu supported versions
#
# Ubuntu release: https://ubuntu.com/about/release-cycle
# bionic=18.04 focal=20.04 jammy=22.04
NBS_MATRIX_UBUNTU_SUPPORTED_VERSIONS=( 'bionic' 'focal' )
#
# iceboxed: implement other OS support (ref task NMO-213 OsX arm64-Darwin and NMO-210 OsX x86 CD components)
#NBS_MATRIX_OSX_SUPPORTED_VERSIONS=( 'monterey' 'ventura' )
NBS_MATRIX_OSX_SUPPORTED_VERSIONS=( )

2 changes: 1 addition & 1 deletion ...ompose_over_build_matrix.bash › build dependencies_arm64v8 ci_PR_arm64v8 _popup_.run.xml
View file
@@ -1,6 +1,6 @@
<component name="ProjectRunConfigurationManager">
<configuration default="false" name="bash lpm_crawl_libpointmatcher_build_matrix.bash › build ci_PR_arm64v8 &lt;popup&gt;" type="ShConfigurationType" folderName="[LPM] C | Execute compose over build matrix" editBeforeRun="true">
<option name="SCRIPT_TEXT" value="bash lpm_crawl_libpointmatcher_build_matrix.bash --repository-version-build-matrix-override latest --os-name-build-matrix-override ubuntu --cmake-build-type-build-matrix-override RelWithDebInfo --ubuntu-version-build-matrix-override focal --fail-fast -- build ci_PR_arm64v8" />
<option name="SCRIPT_TEXT" value="bash lpm_crawl_libpointmatcher_build_matrix.bash --repository-version-build-matrix-override latest --os-name-build-matrix-override ubuntu --cmake-build-type-build-matrix-override Release --ubuntu-version-build-matrix-override focal --fail-fast -- build ci_PR_arm64v8" />
<option name="INDEPENDENT_SCRIPT_PATH" value="true" />
<option name="SCRIPT_PATH" value="" />
<option name="SCRIPT_OPTIONS" value="" />
Expand Down
3 changes: 2 additions & 1 deletion build_system/README.md
View file
@@ -1 +1,2 @@
See [README.md #Contributing](https://github.com/norlab-ulaval/libpointmatcher/tree/master#contributing) for instructions on how integrate the `libpointmatcher-build-system` to your local development workflow (on your workstation).
See [contributing_instructions.md](/doc/contributing/contributing_instructions.md)
for instructions related to bug reporting, code contribution and for setting up the `libpointmatcher-build-system` on your workstation to speed up your local development workflow.

0 comments on commit 65320bc

Please sign in to comment.