refactor(release-prep): doc update and libpointmatcher-build-system logic refactoring to submodule

CHANGELOG.rst

@@ -2,6 +2,91 @@
 Changelog for package libpointmatcher
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
+2.0.0 (2023-12-15)
+-----------------
+* fix: N2ST path resolution in dependencies-doc docker image
+* refactor: move libpointmatcher build-system logic to norlab-build-system submodule
+* fix: build-system side unstable compilation issue fix
+* Fixes to CMake library management config generation
+* New DataPoints filter for descriptor augmentation  Enhancement
+* Fix 2D transformation tests in debug
+* Fix Transformations test build when using Eigen3.4
+* Disable static boost linkage by default
+* Update CompilationPython.md
+* Update README.md
+* Update readme_test.md for docker daemon test
+* Create readme_test.md
+* Updated the inner loop counter
+* build-system minor mod
+* libpointmatcher build-system
+* Fix omega descriptor export
+* Handle libnabo config mode
+* Build python binding as regular package
+* Fix time values after applying Sampling surface normal filter
+* Add seed to Random sampling filter
+* Add more details in pypointmatcher's installation docs
+* Fixed compilation on Visual Studio 2022
+* Fix 4DoF PointToPlane error minimizer crash
+* Use the LOG_INFO_STREAM macro instead of std::cerr
+* package format=3
+* Create LICENSE file based on BSD license as per package.xml
+* catkin not required for pure cmake packages
+* Histogram<T>::computeStats() without sorting the Histogram
+* Windows: Fix Narrowing conversion of seed in MaxPointCount
+* Change storage ordering of the eigen vectors descriptors
+* Apply the transformation to eigen vectors
+* Fix the surface normals datapoints filter covariance matrix bug
+* Fix weird behavior of MaxPointCountDataPointsFilter
+* Bug fix in SurfaceNormalOutlierFilter
+* Update CompilationWindows.md
+* [Matches/OutlierFiltersImpl] Made convergence error messages more informative
+* RemoveBeyond option for the maxQuantileDistance filter
+* refactor: Drop support for yaml-cpp old API  Enhancement
+* Automaticaly find eigen3
+* Strong Windows installation tutorial update
+* Add libpointmatcher_INCLUDE_DIR to match with CGAL
+* Add Boost_LIBRARIES to pointmatcher_LIBRARIES
+* Add option to weight by reading pc
+* Fixes for windows
+* Fix windows doc
+* Include iso646.h to add and, or, not macros
+* Typo in PCL lib name
+* Implemented an in-place method for transforming DataPoints objects
+* The keyword "or" is not supported by windows compilers.
+* Link against pthread
+* [DataPoints] Added check to prevent unsigned int underflow in getEuclideanDim()
+* [feature/spdf] Add SpectralDecompositionFilter (SpDF)
+* Fixed the differences between examples and documentation (#409)
+* Fix build of downstream packages.
+* Reorganization of the compiling tutorials for ubuntu and macos
+* Added reference for PointToPlaneWithCov ErrorMinimizer.
+* [WIP] feature/python_module : Adding libpointmatcher's Python bindings (#222)  Enhancement
+* Feature/speedup random sampling filter
+* Replaced the remaining raw pointers with shared pointers in Registrar.h
+* Adding the new outlierfilters documentation to mkdocs .yml
+* Outlier filter documentation added
+* Fix/typo Fixes tutorials building failure
+* Fixes tutorials formatting problem (#373)
+* [WIP] Fix/typo : Tutorials improvements
+* Fix out-of-bounds access
+* Add missing force4DOF param to PointToPlaneWithCov
+* Modernize cmake; make cmake compatible with git submoduling
+* Feature/4 dof for point to plane minimizer
+* Feature/geometry data points filter for master
+* Get rid of the Visual C++ warnings
+* Change matrix type
+* Update link to documentation in readme
+* Fix segfault happening in ICPSequence class
+* Added missing include that made windows compilation fail.
+* Change icp chain image to an svg
+* Add support for Travis
+* Fix typo
+* Improve speed of Normal Space filtering
+* Fix/normal space hashing
+* Fix/clamp normals
+
+
+
 1.3.1 (2019-03-04)
 ------------------
 * Added documentation for people using ROS.

README.md

@@ -3,14 +3,16 @@
 
 
 ---
-
+`libpointmatcher` is currently tested on our build system with the following architecture and OS
+- x86 and arm64/v8
+- Ubuntu bionic (18.04) and focal (20.04) 
 
 # Documentation and Tutorials
 
 libpointmatcher is a modular library implementing the Iterative Closest Point (ICP) algorithm for aligning point clouds. It has applications in robotics and computer vision.
-The library is written in C++ for effeciency with [bindings in Python](https://github.com/ethz-asl/libpointmatcher/blob/master/doc/index.md#python-).
-
-**Quick link for the tutorial pages: [Tutorials](doc/index.md) (also available on [readthedocs.org](http://libpointmatcher.readthedocs.org/)).**
+The library is written in C++ for effeciency with [bindings in Python](https://github.com/norlab-ulaval/libpointmatcher/blob/master/doc/index.md#python-).
+ 
+**Quick link for the tutorial pages: [Tutorials](doc/index.md)** (also available on [readthedocs.org](http://libpointmatcher.readthedocs.org/) but might not be up-to-date).
 
 Those tutorials are written using Markdown syntax and stored in the project's `/doc` folder.  Their scope ranges from introductory material on performing point cloud registration to instructions for the more experienced developer on how to extend the library's codebase. 
 
@@ -23,6 +25,15 @@ You can read the latest changes in the [release notes](doc/ReleaseNotes.md).
 
 # Quick Start
 
+Clone the repository with its submodule
+```shell
+git clone --recurse-submodules https://github.com/norlab-ulaval/libpointmatcher.git
+
+# If libpointmatcher is already cloned, fetch its new submodule 
+git submodule update --remote --recursive --init
+```
+
+
 Although we suggest to use the [tutorials](doc/index.md), here is a quick version of it:
 
 The library has a light dependency list:
@@ -59,21 +70,101 @@ utest/utest --path ../examples/data/
 
 ### Linking to external projects.
 
-We mainly develop for __cmake projects__ and we provide example files under [`examples/demo_cmake/`](https://github.com/ethz-asl/libpointmatcher/tree/master/examples/demo_cmake) to help you in your own project. We also provide a __QT Creator__ example in [`examples/demo_QT/`](https://github.com/ethz-asl/libpointmatcher/tree/master/examples/demo_Qt), which manually lists all the dependencies in the file [`demo.pro`](https://github.com/ethz-asl/libpointmatcher/blob/master/examples/demo_Qt/demo.pro). You would need to ajust those paths to point at the appropriate locations on your system. For a more detailed procedure, check the [Linking Projects to libpointmatcher](doc/LinkingProjects.md) section.
+We mainly develop for __cmake projects__ and we provide example files under [`examples/demo_cmake/`](https://github.com/norlab-ulaval/libpointmatcher/tree/master/examples/demo_cmake) to help you in your own project. We also provide a __QT Creator__ example in [`examples/demo_QT/`](https://github.com/norlab-ulaval/libpointmatcher/tree/master/examples/demo_Qt), which manually lists all the dependencies in the file [`demo.pro`](https://github.com/norlab-ulaval/libpointmatcher/blob/master/examples/demo_Qt/demo.pro). You would need to ajust those paths to point at the appropriate locations on your system. For a more detailed procedure, check the [Linking Projects to libpointmatcher](doc/LinkingProjects.md) section.
 
+## File formats
 
-### Bug reporting
+The library support different file formats for importing or exporting data:
 
-Please use our [github's issue tracker](http://github.com/ethz-asl/libpointmatcher/issues) to report bugs. If you are running the library on Ubuntu, copy-paste the output of the script [listVersionsUbuntu.sh](https://github.com/ethz-asl/libpointmatcher/blob/master/utest/listVersionsUbuntu.sh) to simplify the search of an answer.
+* csv (Comma Separated Values)
+* vtk (Visualization Toolkit Files)
+* ply (Polygon File Format)
+* pcd (Point Cloud Library Format)
 
-## File formats
-The library support different file formats for importing or exporting data:
-  * csv (Comma Separated Values)
-  * vtk (Visualization Toolkit Files)
-  * ply (Polygon File Format)
-  * pcd (Point Cloud Library Format)
+Those functionnalities are available without increasing the list of dependencies at the expense of
+limited functionality support. For more details, see the
+tutorial [Importing and Exporting Point Clouds](doc/ImportExport.md). Example executables using
+those file formats from the command line can be found in the `/examples` directory and are
+described [here](doc/ICPIntro.md) in more detail.
+
+---
+
+# Contributing
+
+## Bug reporting
+
+Please use our [github's issue tracker](http://github.com/ethz-asl/libpointmatcher/issues) to report bugs. If you are running the library on Ubuntu, copy-paste the output of the script [listVersionsUbuntu.sh](https://github.com/norlab-ulaval/libpointmatcher/blob/master/utest/listVersionsUbuntu.sh) to simplify the search of an answer.
+
+## Codebase
+
+Libpointmatcher codebase now integrate [norlab-build-system (NBS)](https://github.com/norlab-ulaval/norlab-build-system) and [norlab-shell-script-tools (N2ST)](https://github.com/norlab-ulaval/norlab-shell-script-tools). `NBS` is a build-infrastructure-agnostic build system custom-made for our need at NorLab and `N2ST` is library of shell script, function for shell script development and a shell testing tools leveraging `bats-core`.
+
+NBS is deployed on our [TeamCity](https://www.jetbrains.com/teamcity/) continuous integration/deployment server and oversees both protected branch of the [libpointmatcher](https://github.com/norlab-ulaval/libpointmatcher) GitHub repository: the `master` branch and the `develop` branch.
+
+- The `develop` branch can only be merged through a pull-request from any `<feature>` branchs. Any contributor can submit a pull request to the `develop` branch;
+- The `master` branch can only be merged from the `release` branch through a pull-request by a repository admin.
+In both cases submiting a pull request will trigger a build configuration on our build system and the pull request will be granted if the build/test run succeede.
 
-Those functionnalities are available without increasing the list of dependencies at the expense of limited functionality support. For more details, see the tutorial [Importing and Exporting Point Clouds](doc/ImportExport.md). Example executables using those file formats from the command line can be found in the `/examples` directory and are described [here](doc/ICPIntro.md) in more detail.
+**Current build matrix:**
+`[latest] x [x86, arm64] x [ubuntu] x [bionic, focal, jammy] x [Release, RelWithDebInfo, MinSizeRel]`
+
+### Development workflow
+
+To speed up the development process, you can run the build system localy on your workstation and have access to stacktrace and build log. 
+It support multi-OS and multi-architecture through docker container.
+
+#### Install _libpointmatcher-build-system_ dependencies
+```shell
+cd <path/to/libpointmatcher>
+
+# If libpointmatcher is already cloned, fetch the NBS and N2ST submodule 
+git submodule update --remote --recursive --init
+
+cd ./build_system/lpm_utility_script
+
+# Execute docker tools install script i.e. docker daemon, docker compose, docker buildx 
+bash lpm_install_docker_tools.bash
+
+# Configure a multi-architecture docker builder
+bash lpm_create_multiarch_docker_builder.bash
+```
+
+#### libpointmatcher development › to execute build/test step locally
+```shell
+cd <path/to/libpointmatcher>/build_system
+
+# Run the build matrix as specified in ".env.build_matrix.libpointmatcher" on native architecture using "ci_PR" service 
+bash lpm_crawl_libpointmatcher_build_matrix.bash --fail-fast -- build ci_PR
+
+# Run a specific case using build flags with multi-architecture virtualization using "ci_PR_amd64" and "ci_PR_arm64v8" services 
+bash lpm_crawl_libpointmatcher_build_matrix.bash --repository-version-build-matrix-override latest \
+                                                 --os-name-build-matrix-override ubuntu \
+                                                 --cmake-build-type-build-matrix-override RelWithDebInfo \
+                                                 --ubuntu-version-build-matrix-override focal \
+                                                 --fail-fast \
+                                                 -- build ci_PR_amd64 ci_PR_arm64v8
+
+# Read the help for details
+bash lpm_crawl_libpointmatcher_build_matrix.bash --help
+```
+
+#### Build system development › to execute docker dryrun and configuration test
+```shell
+cd <path/to/libpointmatcher>/build_system/tests/
+
+bash run_all_docker_dryrun_and_config_tests.bash
+```
+
+#### Build system notes: 
+- `lpm_crawl_dependencies_build_matrix.bash` execute the build matrix for the libpointmatcher dependencies. 
+   It's not required to build them locally as they are pre-build by our TeamCity server periodically push to dockerhub.
+   When executing `lpm_crawl_libpointmatcher_build_matrix.bash`, the `libpointmatcher-dependencies` docker images are pull and used as base image for the `libpointmatcher-[ci_PR_test|release]` images.
+- About `libpointmatcher/.github/workflow/` vs `libpointmatcher/build_system/` logic: Those are separate build logic.
+  `.github/workflow/` was community contributed and as the responsibilities of building python-binding and pushing packages. 
+  For this reason, it run a one-dimension build matrix: multiple python version, single OS version, single arch (x86) and
+  single compile flag which GitHub action computing resources can handle just fine.
+
+---
 
 # Citing
 

build_system/.env

@@ -8,8 +8,8 @@ NBS_DOCKERHUB_NAMESPACE=norlabulaval
 # Repository url
 #
 # Note:
-#   - Original repo: https://github.com/ethz-asl/libpointmatcher
-#   - DEV repo: https://github.com/norlab-ulaval/libpointmatcher-build-system
+#   - Main repo: https://github.com/norlab-ulaval/libpointmatcher
+#   - Legacy repo: https://github.com/ethz-asl/libpointmatcher
 
 NBS_REPOSITORY_DOMAIN=norlab-ulaval
 

build_system/.jetbrains_run_config/bash lpm_execute_compose_over_build_matrix.bash › build dependencies-general dependencies (BUILD MATRIX SUBSET).run.xml

@@ -1,6 +1,6 @@
 <component name="ProjectRunConfigurationManager">
-  <configuration default="false" name="bash lpm_crawl_dependencies_build_matrix.bash › build dependencies-general dependencies (BUILD MATRIX SUBSET)" type="ShConfigurationType" folderName="(A | Execute compose over build matrix)" editBeforeRun="true">
-    <option name="SCRIPT_TEXT" value="bash lpm_crawl_dependencies_build_matrix.bash --fail-fast --cmake-build-type-build-matrix-override None --os-name-build-matrix-override ubuntu --ubuntu-version-build-matrix-override jammy -- build --push" />
+  <configuration default="false" name="bash lpm_crawl_dependencies_build_matrix.bash › build dependencies-general dependencies (BUILD MATRIX SUBSET)" type="ShConfigurationType" folderName="(A | Execute compose over build matrix)">
+    <option name="SCRIPT_TEXT" value="bash lpm_crawl_dependencies_build_matrix.bash --fail-fast --cmake-build-type-build-matrix-override None --os-name-build-matrix-override ubuntu --ubuntu-version-build-matrix-override focal -- build dependencies-general dependencies dependencies-doc" />
     <option name="INDEPENDENT_SCRIPT_PATH" value="true" />
     <option name="SCRIPT_PATH" value="" />
     <option name="SCRIPT_OPTIONS" value="" />
@@ -16,4 +16,4 @@
     </envs>
     <method v="2" />
   </configuration>
-</component>
+</component>

build_system/.jetbrains_run_config/bash lpm_execute_compose_over_build_matrix.bash › build test_compilation__ (Release).run.xml

@@ -1,5 +1,5 @@
 <component name="ProjectRunConfigurationManager">
-  <configuration default="false" name="bash lpm_crawl_libpointmatcher_build_matrix.bash › build test_compilation_* (Release)" type="ShConfigurationType" folderName="(A | Execute compose over build matrix)">
+  <configuration default="false" name="bash lpm_crawl_libpointmatcher_build_matrix.bash › build test_compilation_* (compilation test)" type="ShConfigurationType" folderName="(A | Execute compose over build matrix)">
     <option name="SCRIPT_TEXT" value="bash lpm_crawl_libpointmatcher_build_matrix.bash --fail-fast --cmake-build-type-build-matrix-override Release --os-name-build-matrix-override ubuntu -- build test_compilation_prefixed_path test_compilation_auto_path_resolution" />
     <option name="INDEPENDENT_SCRIPT_PATH" value="true" />
     <option name="SCRIPT_PATH" value="" />
@@ -16,4 +16,4 @@
     </envs>
     <method v="2" />
   </configuration>
-</component>
+</component>

build_system/lpm_utility_script/lpm_bashrc_config.bash

@@ -19,13 +19,13 @@ function lpm::configure_bashrc() {
   source .env
   set +o allexport
 
-  # ....Helper function..............................................................................................
+  # ....Helper function............................................................................
   # import shell functions from utilities library
   N2ST_PATH=${N2ST_PATH:-"${LPM_PATH}/build_system/utilities/norlab-shell-script-tools"}
   source "${N2ST_PATH}/import_norlab_shell_script_tools_lib.bash"
 
   # ====Begin======================================================================================
-  print_formated_script_header 'lpm_bashrc_config.bash' "${NBS_LINE_CHAR_UTIL}"
+  print_formated_script_header 'lpm_bashrc_config.bash' "${MSG_LINE_CHAR_UTIL}"
 
 
   if [[ "$(basename ${LPM_PATH})" != "${NBS_REPOSITORY_NAME}" ]]; then
@@ -49,7 +49,7 @@ function lpm::configure_bashrc() {
 
   print_msg_done "New aliases with prefix 'lpm' added to .bashrc"
 
-  print_formated_script_footer 'lpm_bashrc_config.bash' "${NBS_LINE_CHAR_UTIL}"
+  print_formated_script_footer 'lpm_bashrc_config.bash' "${MSG_LINE_CHAR_UTIL}"
 
   # ====Teardown===================================================================================
   cd "${TMP_CWD}"

build_system/lpm_utility_script/lpm_create_multiarch_docker_builder.bash

@@ -7,20 +7,19 @@
 #
 
 function lpm::create_multiarch_docker_builder() {
-  # ....Project root logic.........................................................................
   local TMP_CWD
   TMP_CWD=$(pwd)
 
   # ....Project root logic.........................................................................
   LPM_PATH=$(git rev-parse --show-toplevel)
-  cd "${LPM_PATH}/build_system" || exit
 
   # ....Load environment variables from file.......................................................
+  cd "${LPM_PATH}/build_system" || exit
   set -o allexport
   source .env
   set +o allexport
 
-  # ....Helper function..............................................................................................
+  # ....Helper function............................................................................
   # import shell functions from utilities library
   N2ST_PATH=${N2ST_PATH:-"${LPM_PATH}/build_system/utilities/norlab-shell-script-tools"}
   source "${N2ST_PATH}/import_norlab_shell_script_tools_lib.bash"
@@ -42,7 +41,7 @@ function lpm::create_multiarch_docker_builder() {
 
   docker buildx ls
 
-  print_formated_script_footer 'lpm_create_multiarch_docker_builder.bash' "${NBS_LINE_CHAR_UTIL}"
+  print_formated_script_footer 'lpm_create_multiarch_docker_builder.bash' "${MSG_LINE_CHAR_UTIL}"
   # ====Teardown===================================================================================
   cd "${TMP_CWD}"
 }