Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

libpointmatcher build-system #518

Merged
merged 16 commits into from
Aug 9, 2023

Conversation

RedLeader962
Copy link
Collaborator

@RedLeader962 RedLeader962 commented Aug 9, 2023

Implementation of build-system tools for the libpointmatcher codebase

Features:

  • build-infrastructure agnostic tools relying on docker container, docker compose and shell scripts
  • libpointmatcher build matrix:
    • multi-arch support (currently x86 and arm64v8)
    • multi-os (currently only releasing ubuntu but most logic for handling OsX is already done)
    • multi-os version (currently releasing bionic and focal. jammy will require codebase bug fix related to GoogleTest)
    • multi-libpointmatcher release tag
    • multi-libpointmatcher compile flag setting
  • tailor-made build logs to improve dev and debug feedback readability:
    • header, footer and splash screen to quickly identify build step beginning/end
    • two output type: regular consol print and/or TeamCity service message if run on TeamCity server
    • information is structured by build steps: compiling, test, arch, os type, os-version, tag ...
    • meaningful color to highlight code preview and status (warning, passing and failure)
  • Docker images are separated by role:
    • libpointmatcher-dependencies for building dependencies
    • libpointmatcher-dependencies-doc for building documentation related dependencies
    • libpointmatcher-ci-pr for building and testing pull-request with fail-fast policy
    • libpointmatcher-ci-sitrep-matrix for building and testing the complete build matrix despite failure so that we can assess the codebase status
    • libpointmatcher for building release image to Docker Hub
  • several installer scripts are implemented to handle
    • dependencies installation;
    • python version + packages;
    • documentation related dependencies installation;
    • libpointmatcher installation.

Muted codebase failing tests until they can be investigated:

  • Transformation.ComputeRigidTransformDataPoints3D
  • Transformation.ComputeSimilarityTransformDataPoints3D

Example

Case › Build matrix all passed

image

Case › Build matrix sitrep with pass and fail

image


…) › install script + container

- Implemented container+install script with libpointmatcher dependencies (Ubuntu x86 case);
- Implemented libpointmatcher container+install script (Ubuntu x86 case);
- Fixed libpointmatcher install script + dockerfile general case (build OK, but utest fail 2 tests);
- Added debug mode to `lpm_execute_compose.bash`;
- Fix dependencies tools: install script and dockerfile;
- Implemented build matrix crawler;
- Implemented test tools for build system user script dev and test; Finish implementing lpm_build_and_run_container.bash;
- Finish implementing the build dependencies step: `lpm_install_dependencies_ubuntu.bash` + docker compose for build;
…stall script;

Task:
- NMO-249 ⚒︎ → Pre-condition test validating that variable in .env are exported
- NMO-270 ♻︎ → Solution for texlive-full package 6GB size
- NMO-264 LPM build case › generate doc
- NMO-260 ⚒︎ → Step › build and push all LPM dependencies images
- NMO-261 ⚒︎ → Step › build Libpointmatcher (checkout PR branch) and run 'entrypoint_execute_lpm_unittest.bash'
- NMO-262 ⚒︎ → Build and push all Libpointmatcher release images
…plash screen, fixed print terminal wide line to separate clearly execution steps;
- Fix "tput: No value for $TERM and no -T specified" error
- Add splash screen consol print to improve dev and debug feedback readability;
- Remove the shebang interactive flag
…status

- Mute failing tests:
   - Transformation.ComputeRigidTransformDataPoints3D,
   - Transformation.ComputeSimilarityTransformDataPoints3D
- Test TeamCity error collection behaviour ›  exit with error cases (remove)
- fix xterm var problem
- Mute install step temporary;
- Add LPM_IMAGE_ARCHITECTURE logic to lpm_install_libpointmatcher_ubuntu.bash;
- Fix missing copy file in dockerfile;
- Added the LPM_IMAGE_ARCHITECTURE env var in the header/footer;
- Added the platform arg in compose;
- Added the platform arg in compose and remove arch info from tag;
…un test per aarch step

- muted --job-id flag; cleanup comment;
- mod compose
- priviledge off
- mod compose service "ci_build_and_test" + refactor Dockerfile.libpointmatcher_ci_build_and_test
- implement compose service "ci_build_and_test";
- revert entrypoint;
- fix ci dockerfile;
- added job ID flag;
- mod compose logic;
- fix compose file;
- mod shebang;
- fix the prinf -<char> invalide option error;
- fix draw_horizontal_line_across_the_terminal_window function;
- Teamcity build test: mod dockerfile to first build install script then run test (instead of run install and test);
- Teamcity build test: docker compose mod;
- tput fix for teamcity;
- entrypoint mod to exec;
- Multi-arch run ci service logic;
- Test split build run by arch (case dependencies multi-arch) › test split by install script steps;
- TeamCity service message logic
   - Add TeamCity bloc service message logic (with arch version) to dependencies installer script
   - Add TeamCity compilation service message logic to dependencies installer script
- Mod .env.prompt
- TEST TEAMCITY_* environment variable access;
- Remove ubuntu 22.10 'kinetic' and ubuntu 23.04 'lunar' from the build matrix;
- TeamCity build config test › Build matrix dependencies image build
- Mod console print feedback for both execute compose script
- TeamCity build configuration › trigger
- Added TeamCity service message logic to improve readability by structuring build log steps:
    - Implemented a function which either start or finish TeamCity **compilation** service message
    - Implemented a function which either opened or closed TeamCity **bloc** service message
    - Refactored the lpm_install_dependencies_ubuntu.bash script with both service msg functions
- Added docker debug flag in lpm_execute_compose_over_build_matrix.bash;
- Added docker build arg LIBPOINTMATCHER_CMAKE_BUILD_TYPE;
- Mod splash screen console print alignment;
- NMO-275 Validate that 'utest/listVersionsUbuntu.sh' can run inside container
- Fixed the cmake step typo in installer;
- Add tty and stcin_open flag to each compose service to mitigate log coloring issue;
- Refactor lpm_install_libpointmatcher_ubuntu.bash with TeamCity service message functionalities (DEBUG)
- Fixed ca-certificat related issue
- Fix ubuntu distro condition;
- Refactored lpm_install_doc_dependencies_ubuntu.bash with Teamcity service message functionalities;
- Fix pip install related steps in lpm_install_python_dev_tools.bash;
- Refactor lpm_install_python_dev_tools.bash install steps organization;
- Switch bash command to source in dockerfile RUN step so that stdout/stderr can be writen in TeamCity build logs;
- Added a conditional step so that lpm_install_python_dev_tools.bash is only executed once when the script is called inside a docker build;
- Refactor Dockerfile.dependencies to be able to run lpm_install_python_dev_tools.bash;
- Refactor lpm_install_python_dev_tools.bash with new TeamCity service msg functionalities;
- Updated doc;
…full repo build status

- Added CMAKE_BUILD_TYPE to the build matrix
   - implemented the CMAKE_BUILD_TYPE build matrix crawl logic
   - implemented TeamCity test service message error reporting
   - implemented logic to propagate error exit code from script executed in docker command out to lpm_execute_compose_over_build_matrix.bash
   - added a `--fail-fast flag`
   - added a `--libpointmatcher-cmake-build-type-build-matrix-override`
   - refactored ``--debug` to ``--docker-debug-logs`
- fix path to file typo in compose file; switch build matrix order
- Fixed git checkout logic for pulling specific branch/tag;
- Implemented a dockerfile specific for repo sitrep run;
- Refactored docker compose service name to be less verbose;
- Added logic for configuring ci-sitrep run vs PR run;
- Fixed lpm_execute_compose_over_build_matrix.bash error exit code; Fixed teamcity test service message;
- Added Teamcity bloc open/closed service message to build matrix steps: os name , version, libpointmatcher version
- fix path to build dir;
- fix listVersionUbuntu.sh path;
- set both CMAKE_INSTALL_PREFIX and LIBNABO_INSTALL_DIR cmake flag;
- Added repository test tag: v2.0.test
@ethzasl-jenkins
Copy link

Can one of the admins verify this patch?

@RedLeader962 RedLeader962 marked this pull request as ready for review August 9, 2023 00:40
@pomerlef
Copy link
Collaborator

pomerlef commented Aug 9, 2023

We are depreciating ethzasl-jenkins and changing the CI/CD framework from libpointmatcher. Any comments are welcome, otherwise, we merge in a day!

@pomerlef pomerlef merged commit cba5918 into norlab-ulaval:master Aug 9, 2023
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants