Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add dockerhub release logic and improve main readme #550

Merged
merged 14 commits into from Jan 18, 2024
Merged
Show file tree
Hide file tree
Changes from 8 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
23 changes: 5 additions & 18 deletions .github/pull_request_template.md
Expand Up @@ -2,6 +2,7 @@

### Summary:


### Changes and type of changes (quick overview):

-
Expand All @@ -19,7 +20,7 @@
- [ ] I have commented hard-to-understand code
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] All tests pass locally with my changes
(Check [README.md #Contributing](https://github.com/norlab-ulaval/libpointmatcher/tree/develop#contributing)
(Check [contributing_instructions.md](https://github.com/norlab-ulaval/libnabo/tree/develop/doc/contributing/contributing_instructions.md)
boxanm marked this conversation as resolved.
Show resolved Hide resolved
for local testing procedure using _libpointmatcher-build-system_)

### PR creation related
Expand All @@ -33,22 +34,8 @@

- [ ] I have included a quick summary of the changes
- [ ] I have indicated the related issue's id with `# <issue-id>` if changes are of type `fix`
- [ ] I have included a high-level list of changes and their corresponding type
- Types: `feat` `fix` `docs` `style` `refactor` `perf` `test` `build` `ci` `chore` `revert`
- Breaking changes: `<type>!`
- Reference:
see [commit_msg_reference.md](https://github.com/norlab-ulaval/libpointmatcher/blob/develop/commit_msg_reference.md)
in the repository root for details
- [ ] I have included a high-level list of changes and their corresponding types
(See [commit_msg_reference.md](https://github.com/norlab-ulaval/libnabo/tree/develop/doc/contributing/commit_msg_reference.md)
boxanm marked this conversation as resolved.
Show resolved Hide resolved
for details)

---

## Note for repository admins

### Release PR related

- Only repository admins have the privilege to `push/merge` on the default branch (ie: `master`)
and the `release` branch.
- Keep PR in `draft` mode until all the release reviewers are ready to push the release.
- Once a PR from `release` -> `master` branch is created (not in draft mode),
- it triggers the _build-system_ test
- (in-progress) and it triggers the _semantic release automation_
5 changes: 2 additions & 3 deletions .github/workflows/build-python.yaml
Expand Up @@ -177,7 +177,7 @@ jobs:
if: ${{ steps.cache-boost.outputs.cache-hit != 'true' && runner.os == 'Linux' }}
working-directory: ${{ env.BOOST_SRC_DIR }}
run: |
wget --no-verbose https://boostorg.jfrog.io/artifactory/main/release/${{ env.BOOST_VERSION }}/source/${{ env.BOOST_ARCHIVE_NAME }}
wget --no-verbose https://archives.boost.io/release/${{ env.BOOST_VERSION }}/source/${{ env.BOOST_ARCHIVE_NAME }}
7z -y x ${{ env.BOOST_ARCHIVE_NAME }}
rm ${{ env.BOOST_ARCHIVE_NAME }}

Expand All @@ -192,7 +192,7 @@ jobs:
if: steps.cache-libnabo.outputs.cache-hit != 'true'
working-directory: ${{ env.LIBNABO_SRC_DIR }}
run: |
git clone -b ${{ env.LIBNABO_VERSION }} --single-branch https://github.com/ethz-asl/libnabo.git
git clone -b ${{ env.LIBNABO_VERSION }} --single-branch https://github.com/norlab-ulaval/libnabo.git

- name: Install libnabo ${{ env.LIBNABO_VERSION }} on Windows
if: ${{ steps.cache-libnabo.outputs.cache-hit != 'true' && runner.os == 'Windows' }}
Expand Down Expand Up @@ -319,4 +319,3 @@ jobs:
prerelease: true
allowUpdates: true
generateReleaseNotes: true

178 changes: 65 additions & 113 deletions README.md
Expand Up @@ -2,6 +2,25 @@

![banner](doc/images/banner_light.jpeg)

[//]: # ( ==== Description =========================================== )
**_libpointmatcher_ is a modular library implementing the Iterative Closest Point (ICP) algorithm<br>
for aligning point clouds. It has applications in robotics and computer vision.**
<br>
The library is written in C++ for effeciency
with [bindings in Python](doc/index.md#python-).
<br>
<br>

[//]: # (====GitHub badges========================================================================)

<img alt="GitHub Repo stars" src="https://img.shields.io/github/stars/norlab-ulaval/libpointmatcher">
<img alt="GitHub forks" src="https://img.shields.io/github/forks/norlab-ulaval/libpointmatcher">
<img alt="GitHub License" src="https://img.shields.io/github/license/norlab-ulaval/libpointmatcher">
<img alt="GitHub release (with filter)" src="https://img.shields.io/github/v/release/norlab-ulaval/libpointmatcher">
<img src="https://img.shields.io/static/v1?label=JetBrains TeamCity&message=CI/CD&color=green?style=plastic&logo=teamcity" />
<br>
<br>

[//]: # (====Awesome badges=======================================================================)

[![Mentioned in Awesome LIDAR](https://awesome.re/mentioned-badge.svg)](https://github.com/szenergy/awesome-lidar#basic-matching-algorithms)
Expand All @@ -19,35 +38,27 @@
<a href="http://jslee02.github.io/awesome-robotics-libraries/#3d-mapping">Robotics Libraries</a>
</sup>


[//]: # (====GitHub badges========================================================================)

<img alt="GitHub Repo stars" src="https://img.shields.io/github/stars/norlab-ulaval/libpointmatcher">
<img alt="GitHub forks" src="https://img.shields.io/github/forks/norlab-ulaval/libpointmatcher">
<img alt="GitHub License" src="https://img.shields.io/github/license/norlab-ulaval/libpointmatcher">
<img alt="GitHub release (with filter)" src="https://img.shields.io/github/v/release/norlab-ulaval/libpointmatcher">
<img src="https://img.shields.io/static/v1?label=JetBrains TeamCity&message=CI/CD&color=green?style=plastic&logo=teamcity" />
<br>
<br>
</div>



[//]: # (====Supported OS and aarch===============================================================)

`libpointmatcher` is tested on our build system under the following architecture and OS:
- x86 and arm64/v8
### Supported OS And Architecture
_libpointmatcher_ is tested on our build system under the following architecture and OS:
- Ubuntu bionic (18.04) and focal (20.04)
- x86 and arm64/v8

Note:
- support for Ubuntu jammy (22.04) comming soon
- `libpointmatcher` reportedly works on MacOs OsX (latest) and Windows (latest)
- _libpointmatcher_ reportedly works on MacOs OsX (latest) and Windows (latest)


---

[//]: # (====Release note=========================================================================)

### ★ Version `1.4.0` release note (important)
### ★ Version `1.4.0` Release Note
This release of _libpointmatcher_ introduces the integration of [norlab-build-system (NBS)](https://github.com/norlab-ulaval/norlab-build-system) as a _git submodule_ for codebase development and testing.

Execute the following to clone the repository with its submodule:
Expand All @@ -59,11 +70,14 @@ If _libpointmatcher_ was previously cloned, execute the following to fetch its n
git submodule update --remote --recursive --init
```

### ★ Contributing Instructions
See [contributing_instructions.md](doc/contributing/contributing_instructions.md)
for instructions related to bug reporting, code contribution and for setting up the `libpointmatcher-build-system`
on your workstation to speed up your local development workflow.


[//]: # (====Body=================================================================================)
# Documentation and Tutorials

libpointmatcher is a modular library implementing the Iterative Closest Point (ICP) algorithm for aligning point clouds. It has applications in robotics and computer vision.
The library is written in C++ for effeciency with [bindings in Python](https://github.com/norlab-ulaval/libpointmatcher/blob/master/doc/index.md#python-).

**Quick link for the tutorial pages: [Tutorials](doc/index.md)** (also available on [readthedocs.org](http://libpointmatcher.readthedocs.org/) but might not be up-to-date).

Expand Down Expand Up @@ -93,9 +107,41 @@ and was compiled on:
* Mac OS X ([see how](/doc/CompilationMac.md))
* Windows ([see how](/doc/CompilationWindows.md) - partially supported)

### Docker images

Run the following commands to pull and run libnabo in a docker container
RedLeader962 marked this conversation as resolved.
Show resolved Hide resolved

```shell
docker pull norlabulaval/libpointmatcher:latest-ubuntu-focal

docker run -it --rm norlabulaval/libpointmatcher:latest-ubuntu-focal
```

See
available [libpointmatcher image tags](https://hub.docker.com/repository/docker/norlabulaval/libpointmatcher/)
boxanm marked this conversation as resolved.
Show resolved Hide resolved
on dockerhub.

To install docker related dependencies on ubuntu, execute the following
```shell
cd ./build_system/lpm_utility_script

# Execute docker tools install script i.e. docker daemon, docker compose, docker buildx
bash lpm_install_docker_tools.bash
```


### Compilation & Installation

For beginner users unfamiliar with compiling and installing a library in Linux, go [here](doc/CompilationUbuntu.md) for detailed instructions on compiling libpointmatcher from the source code.

For conveniences, you can use the provided installer script for ubuntu
```shell
bash libpointmatcher_dependencies_installer.bash

# Use the --help flag to see the list of optional flag
bash libpointmatcher_installer.bash [<optional flag>]
```

If you are comfortable with Linux and CMake and have already installed the prerequisites above, the following commands should install libpointmatcher on your system.

```bash
Expand All @@ -105,6 +151,7 @@ make
sudo make install
```


### Testing

Libpointmatcher ships with a version of the Google testing framework [GTest](https://github.com/google/googletest). Unit tests are located in the `utest/` directory and are compiled with libpointmatcher (CMake variable `BUILD_TESTS` must be set to `TRUE` before compiling). To run the tests and make sure that your compiled version is working correctly, run the test executable in your build directory:
Expand Down Expand Up @@ -133,101 +180,6 @@ tutorial [Importing and Exporting Point Clouds](doc/ImportExport.md). Example ex
those file formats from the command line can be found in the `/examples` directory and are
described [here](doc/ICPIntro.md) in more detail.

---

# Contributing

## Bug reporting

Please use our [github's issue tracker](http://github.com/ethz-asl/libpointmatcher/issues) to report bugs. If you are running the library on Ubuntu, copy-paste the output of the script [listVersionsUbuntu.sh](https://github.com/norlab-ulaval/libpointmatcher/blob/master/utest/listVersionsUbuntu.sh) to simplify the search of an answer.

## Codebase development

Libpointmatcher codebase now integrate [norlab-build-system (NBS)](https://github.com/norlab-ulaval/norlab-build-system) and [norlab-shell-script-tools (N2ST)](https://github.com/norlab-ulaval/norlab-shell-script-tools).
`NBS` is a build-infrastructure-agnostic build system custom-made to meet our needs in robotic software engineering at NorLab and `N2ST` is a library of shell script functions as well as a shell testing tools leveraging _**bats-core**_ and _**docker**_ .
`N2ST` purpose is to speed up shell script development and improve reliability.

`NBS` is deployed on our [TeamCity](https://www.jetbrains.com/teamcity/) continuous integration/deployment server and oversees protected branches of the [libpointmatcher](https://github.com/norlab-ulaval/libpointmatcher) GitHub repository:

- The `develop` branch can only be merged through a pull-request from any `<feature>` branches. Any contributor can submit a pull request to the `develop` branch;
- the `release` branch is a revision and preparation branch where we can freeze the codebase in a given state without stalling to `develop` branch progression;
- The `master` branch can only be merged through a pull-request from the `release` branch. Only repository admin can submit a PR to the `master` branch.

In any cases, submitting a pull request to `develop` or `master` will trigger a build/test configuration on our build system and the pull request will be granted if the build/test run succeed.

**Current build matrix:**
`[latest] x [x86, arm64] x [ubuntu] x [bionic, focal] x [Release, RelWithDebInfo, MinSizeRel]`

### Development workflow

To speed up the development process, you can run the build system localy on your workstation and have access to stacktrace and build log.
It support multi-OS and multi-architecture through docker container.

#### Install _libpointmatcher-build-system_ dependencies
```shell
cd <path/to/libpointmatcher>

# If libpointmatcher is already cloned, fetch the NBS and N2ST submodule
git submodule update --remote --recursive --init

cd ./build_system/lpm_utility_script

# Execute docker tools install script i.e. docker daemon, docker compose, docker buildx
bash lpm_install_docker_tools.bash

# Configure a multi-architecture docker builder
bash lpm_create_multiarch_docker_builder.bash
```

#### libpointmatcher development › to execute build/test step locally
```shell
cd <path/to/libpointmatcher>/build_system

# Run the build matrix as specified in ".env.build_matrix.libpointmatcher"
# on native architecture using "ci_PR" service
bash lpm_crawl_libpointmatcher_build_matrix.bash --fail-fast -- build ci_PR

# Run a specific case using build flags with multi-architecture
# virtualization using "ci_PR_amd64" and "ci_PR_arm64v8" services
bash lpm_crawl_libpointmatcher_build_matrix.bash \
--repository-version-build-matrix-override latest \
--os-name-build-matrix-override ubuntu \
--cmake-build-type-build-matrix-override RelWithDebInfo \
--ubuntu-version-build-matrix-override focal \
--fail-fast \
-- build ci_PR_amd64 ci_PR_arm64v8

# Read the help for details
bash lpm_crawl_libpointmatcher_build_matrix.bash --help
```
Note: To assess the state of the codebase, even for cases that are known the break the build,
execute `lpm_crawl_libpointmatcher_build_matrix.bleeding.bash` with build matrix `.env.build_matrix.libpointmatcher.bleeding`.
The stable build matrix used for release is `.env.build_matrix.libpointmatcher`.


#### Build system development
```shell
cd <path/to/libpointmatcher>/build_system/tests/

# To execute docker dryrun and configuration tests
bash run_all_docker_dryrun_and_config_tests.bash

# To execute shell script tests
bash run_bats_core_test_in_n2st.bash

# To spin a container in interactive mode with the codebase cloned but not compiled
cd ./tests_docker_interactive/
bash build_and_run_IamBuildSystemTester.bash bash
```

#### Build system notes:
- `lpm_crawl_dependencies_build_matrix.bash` execute the build matrix for the libpointmatcher dependencies.
It's not required to build them locally as they are pre-build by our TeamCity server periodically push to dockerhub.
When executing `lpm_crawl_libpointmatcher_build_matrix.bash`, the `libpointmatcher-dependencies` docker images are pull and used as base image for the `libpointmatcher-[ci_PR_test|release]` images.
- About `libpointmatcher/.github/workflow/` vs `libpointmatcher/build_system/` logic: Those are separate build logic.
`.github/workflow/` was community contributed and as the responsibilities of building python-binding and pushing packages.
For this reason, it run a one-dimension build matrix: multiple python version, single OS version, single arch (x86) and
single compile flag which GitHub action computing resources can handle just fine.

---

Expand Down Expand Up @@ -301,7 +253,7 @@ libpointmatcher is released under a permissive BSD license. Enjoy!
[CMake documentation]: https://cmake.org/cmake/help/v3.10/
[git]: http://git-scm.com
[Eigen]: http://eigen.tuxfamily.org
[libnabo]: http://github.com/ethz-asl/libnabo
[libnabo]: https://github.com/norlab-ulaval/libnabo
[ROS]: http://www.ros.org/
[Paraview]: http://www.paraview.org/
[yaml-cpp]: https://github.com/jbeder/yaml-cpp
Expand Down
3 changes: 1 addition & 2 deletions build_system/.env.build_matrix.dependencies
Expand Up @@ -11,8 +11,7 @@ NBS_EXECUTE_BUILD_MATRIX_OVER_COMPOSE_FILE=docker-compose.dependencies.yaml
# Libpointmatcher version
#
# 'latest' is the latest push to the libpointmatcher master branch
#NBS_MATRIX_REPOSITORY_VERSIONS=( 'v1.3.1' 'latest' 'v2.0.test' )
NBS_MATRIX_REPOSITORY_VERSIONS=( 'latest' )
NBS_MATRIX_REPOSITORY_VERSIONS=( '1.4.0' 'latest' )

#
# Libpointmatcher dependencies CMAKE_BUILD_TYPE
Expand Down
2 changes: 1 addition & 1 deletion build_system/.env.build_matrix.libpointmatcher
Expand Up @@ -11,7 +11,7 @@ NBS_EXECUTE_BUILD_MATRIX_OVER_COMPOSE_FILE=docker-compose.libpointmatcher.yaml
# Libpointmatcher version
#
# 'latest' is the latest push to the libpointmatcher master branch
#NBS_MATRIX_REPOSITORY_VERSIONS=( 'v1.3.1' 'latest' 'v2.0.test' )
#NBS_MATRIX_REPOSITORY_VERSIONS=( '1.4.0' 'latest' )
NBS_MATRIX_REPOSITORY_VERSIONS=( 'latest' )

#
Expand Down
2 changes: 1 addition & 1 deletion build_system/.env.build_matrix.libpointmatcher.bleeding
Expand Up @@ -11,7 +11,7 @@ NBS_EXECUTE_BUILD_MATRIX_OVER_COMPOSE_FILE=docker-compose.libpointmatcher.yaml
# Libpointmatcher version
#
# 'latest' is the latest push to the libpointmatcher master branch
#NBS_MATRIX_REPOSITORY_VERSIONS=( 'v1.3.1' 'latest' 'v2.0.test' )
#NBS_MATRIX_REPOSITORY_VERSIONS=( '1.4.0' 'latest' )
NBS_MATRIX_REPOSITORY_VERSIONS=( 'latest' )

#
Expand Down
38 changes: 38 additions & 0 deletions build_system/.env.build_matrix.libpointmatcher.release
@@ -0,0 +1,38 @@
#
# Build matrix variables
#

#
# The compose file on which the build matrix will be crawled
#
NBS_EXECUTE_BUILD_MATRIX_OVER_COMPOSE_FILE=docker-compose.libpointmatcher.yaml

#
# Libpointmatcher version
#
# 'latest' is the latest push to the libpointmatcher master branch
NBS_MATRIX_REPOSITORY_VERSIONS=( '1.4.0' 'latest' )

#
# Libpointmatcher CMAKE_BUILD_TYPE
#
NBS_MATRIX_CMAKE_BUILD_TYPE=( 'Release' )

#
# LIBPOINTMATCHER supported OS
#
# ToDo: implement OsX support for arm64-Darwin (ref task NMO-213)
#NBS_MATRIX_SUPPORTED_OS=( 'ubuntu' 'l4t' 'osx' )
NBS_MATRIX_SUPPORTED_OS=( 'ubuntu' )

#
# ubuntu supported versions
#
# Ubuntu release: https://ubuntu.com/about/release-cycle
# bionic=18.04 focal=20.04 jammy=22.04
NBS_MATRIX_UBUNTU_SUPPORTED_VERSIONS=( 'bionic' 'focal' )
#
# iceboxed: implement other OS support (ref task NMO-213 OsX arm64-Darwin and NMO-210 OsX x86 CD components)
#NBS_MATRIX_OSX_SUPPORTED_VERSIONS=( 'monterey' 'ventura' )
NBS_MATRIX_OSX_SUPPORTED_VERSIONS=( )

3 changes: 2 additions & 1 deletion build_system/README.md
@@ -1 +1,2 @@
See [README.md #Contributing](https://github.com/norlab-ulaval/libpointmatcher/tree/master#contributing) for instructions on how integrate the `libpointmatcher-build-system` to your local development workflow (on your workstation).
See [contributing_instructions.md](https://github.com/norlab-ulaval/libnabo/tree/develop/doc/contributing/contributing_instructions.md)
boxanm marked this conversation as resolved.
Show resolved Hide resolved
for instructions related to bug reporting, code contribution and for setting up the `libpointmatcher-build-system` on your workstation to speed up your local development workflow.