Merge pull request #2496 from GMLC-TDC/develop

Develop

.github/RELEASE_CHECKLIST_TEMPLATE.md

@@ -0,0 +1,35 @@
+- [ ] Check that all expected [release assets](https://github.com/GMLC-TDC/HELICS/releases/${HELICS_VERSION}) were compiled for all platforms, the `*-source.tar.gz` file, and the `*-SHA-256.txt` file; if any are missing see `Steps to manually add a missing asset to a release` below
+- [ ] Make sure <https://github.com/GMLC-TDC/helics-packaging/blob/main/HELICS_VERSION> has been updated with the new version number; if it hasn't, manually update it (and double check that all release assets were successfully compiled)
+
+- [ ] Update the helics-apps PyPI package by [creating a new release](https://github.com/GMLC-TDC/helics-packaging/releases/new?tag=${HELICS_VERSION}&title=HELICS%20${HELICS_VERSION}%20release&body=HELICS%20${HELICS_VERSION}%20release%20changes:%20https://github.com/GMLC-TDC/HELICS/releases/tag/${HELICS_VERSION}) with the new version tag, title, and description similar to past releases.
+  - [ ] Verify that the [Build Python Packages workflow](https://github.com/GMLC-TDC/helics-packaging/actions/workflows/pythonpackage.yml) for the release (not tag) runs to completion
+  - [ ] Verify the files uploaded to PyPI at <https://pypi.org/project/helics-apps/#files> include wheel files for win32, win_amd64, macosx, and manylinux
+- [ ] Run the [Update HELICS Packages workflow](https://github.com/GMLC-TDC/helics-packaging/actions/workflows/update-helics.yml)
+  - [ ] Verify that a PR was opened at <https://github.com/spack/spack/pulls> that looks okay and leave a review approving it. If the workflow failed or the package PR has CI errors, debug and fix
+  - [ ] Verify that a PR was opened at <https://github.com/msys2/MINGW-packages/pulls> that looks okay and leave a review approving it. If the workflow failed or the package PR has CI build errors, debug and fix
+  - [ ] Verify that a PR was opened at <https://github.com/JuliaPackaging/Yggdrasil/pulls> that looks okay and leave a review approving it. If the workflow failed or the package PR has CI build errors, debug and fix
+  - [ ] Verify [pyhelics](https://github.com/GMLC-TDC/pyhelics) has been updated to include new/removed features, then [create a new release](https://github.com/GMLC-TDC/pyhelics/releases/new?tag=${HELICS_VERSION}&title=${HELICS_VERSION}&body=**Full%20Changelog**:%20https://github.com/GMLC-TDC/pyhelics/compare/${HELICS_PREV_VERSION}...${HELICS_VERSION}) with a matching version tag, and link to a changelog showing the difference between the two versions
+  - [ ] Regenerate the [matHELICS](https://github.com/GMLC-TDC/matHELICS) bindings using the C header file from HELICS, then [create a new release](<https://github.com/GMLC-TDC/matHELICS/releases/new?tag=${HELICS_VERSION}&title=${HELICS_VERSION}&body=Release%20for%20[HELICS%20${HELICS_VERSION}](https://github.com/GMLC-TDC/HELICS/releases/tag/${HELICS_VERSION})>) with a version tag and link to the HELICS release notes.
+- [ ] Run [helics-ns3 CI tests](https://github.com/GMLC-TDC/helics-ns3/actions/workflows/ci.yml) and fix any issues
+- [ ] Run [Test HELICS Install Action](https://github.com/GMLC-TDC/helics-action/actions/workflows/test-helics-install.yml) with both binary and source options selected; fix any issues
+
+---
+
+If part of the [Update HELICS Packages workflow](https://github.com/GMLC-TDC/helics-packaging/actions/workflows/update-helics.yml) fails, the failing part can be re-run on its own:
+
+- [Spack Package Update workflow](https://github.com/GMLC-TDC/helics-packaging/actions/workflows/update-spack-package.yml)
+- [MINGW Package Update workflow](https://github.com/GMLC-TDC/helics-packaging/actions/workflows/update-mingw-package.yml)
+- [Julia/Yggdrasil Package Update workflow](https://github.com/GMLC-TDC/helics-packaging/actions/workflows/update-yggdrasil-package.yml)
+- [pyhelics Update HELICS Version workflow](https://github.com/GMLC-TDC/pyhelics/actions/workflows/update-helics.yml)
+- [matHELICS Update HELICS Version workflow](https://github.com/GMLC-TDC/matHELICS/actions/workflows/update-helics.yml)
+
+---
+
+Steps to manually add a missing asset to a release:
+
+1. If it was a temporary issue such as a resource being unavailable, manually trigger the [release build workflow](https://github.com/GMLC-TDC/HELICS/actions/workflows/release-build.yml) **using the workflow from `Tag: ${HELICS_VERSION}`**.
+2. If it was a more major issue, create a branch based on the tagged release and modify the workflow and scripts in the .github subfolder as needed to make it build. Manually trigger the [release build workflow](https://github.com/GMLC-TDC/HELICS/actions/workflows/release-build.yml) **using the workflow from the branch with these fixes** and for the commit-ish field enter the release tag **${HELICS_VERSION}**.
+3. Download the artifact from the workflow re-run that was missing and upload it to the release.
+4. Download the SHA-256.txt file for the release and add an entry for the new asset (run the `sha256sum` command on Linux to get the line to add). Replace the old SHA-256.txt file for the release with the updated version.
+
+@GMLC-TDC/helics-releases check release tasks related to interfaces/repositories you work on.

.github/actions/docker/develop_docker.sh

@@ -3,4 +3,4 @@
 docker build . -f config/Docker/Dockerfile-HELICS-apps --tag "helics/helics:develop" || rv=1
 docker push "helics/helics:develop" || rv=1
 
-exit $rv
+exit "${rv= 0}"

.github/actions/docker/main_docker.sh

@@ -6,4 +6,4 @@ docker push "helics/helics:latest" || rv=1
 docker build . -f config/Docker/Dockerfile-HELICS-builder --tag "helics/helics:builder" || rv=1
 docker push "helics/helics:builder" || rv=1
 
-exit $rv
+exit "${rv= 0}"

.github/actions/docker/release_docker.sh

@@ -8,4 +8,4 @@ fi
 docker build . -f config/Docker/Dockerfile-HELICS-apps --tag "helics/helics:$1" || rv=1
 docker push "helics/helics:$1" || rv=1
 
-exit $rv
+exit "${rv= 0}"

.github/actions/docker/test_docker.sh

@@ -9,4 +9,4 @@ docker push "helics/helics:test-builder" || rv=1
 docker build . -f config/Docker/Dockerfile-HELICS --tag "helics/helics:test" || rv=1
 docker push "helics/helics:test" || rv=1
 
-exit $rv
+exit "${rv= 0}"

.github/dependabot.yml

@@ -0,0 +1,8 @@
+version: 2
+updates:
+  - package-ecosystem: 'github-actions'
+    directory: '/'
+    schedule:
+      interval: 'weekly'
+    target-branch: 'develop'
+    open-pull-requests-limit: 10

.github/workflows/create-release-checklist.yml

@@ -0,0 +1,45 @@
+name: Create release checklist
+
+on:
+  # Picking a workflow branch that is a tag, in combination with using the commit field is likely to fail for magic git clone/checkout reasons
+  workflow_dispatch:
+    inputs:
+      version:
+        description: 'Version tag to open the release checklist for (including `v` prefix, e.g. `v3.1.0`)'
+        required: true
+        type: string
+  workflow_call:
+    inputs:
+      version:
+        required: true
+        type: string
+  release:
+    types: published
+
+jobs:
+  create-release-checklist-issue:
+    name: Open release checklist issue
+    runs-on: ubuntu-latest
+    steps:
+    # Needed to get the checklist template
+    - name: Checkout release checklist template
+      uses: actions/checkout@v3
+    - name: Create issue
+      env:
+        GH_TOKEN: ${{ github.token }}
+      run: |
+        echo "$PWD"
+        if [[ "${{ inputs.version }}" ]]; then
+          HELICS_VERSION="${{ inputs.version }}"
+        else
+          HELICS_VERSION="${{ github.event.release.tag_name }}"
+        fi
+        export HELICS_VERSION
+        echo "HELICS_VERSION=${HELICS_VERSION}"
+        HELICS_VERSION_MAJ="$(echo "${HELICS_VERSION}" | cut -d "." --fields=1)"
+        HELICS_VERSION_MAJ_MIN="$(echo "${HELICS_VERSION}" | cut -d "." --fields=1,2)"
+        HELICS_PREV_VERSION=$(gh release list --limit 100 | grep -v "Pre-release" | cut -f3 | grep "${HELICS_VERSION_MAJ}" | head -n2 | tail -n1)
+        export HELICS_PREV_VERSION
+        echo "HELICS_PREV_VERSION=${HELICS_PREV_VERSION}"
+        RELEASE_CHECKLIST="$(envsubst < ./.github/RELEASE_CHECKLIST_TEMPLATE.md)"
+        gh issue create --title "${HELICS_VERSION} Release Checklist" --body "${RELEASE_CHECKLIST}"

.github/workflows/docker.yml

@@ -36,10 +36,10 @@ jobs:
       packages: write
 
     steps:
-    - uses: actions/checkout@v1
+    - uses: actions/checkout@v3
     - name: Docker meta data
       id: meta
-      uses: docker/metadata-action@v3
+      uses: docker/metadata-action@v4
       with:
         images: |
           helics/helics
@@ -56,13 +56,13 @@ jobs:
           type=semver,pattern={{version}}
     - name: Login to Docker Hub
       if: github.event_name != 'workflow_dispatch' || !github.event.inputs.build_only
-      uses: docker/login-action@v1
+      uses: docker/login-action@v2
       with:
         username: ${{ secrets.DOCKER_USERNAME }}
         password: ${{ secrets.DOCKER_PASSWORD }}
     - name: Login to GitHub Container Registry
       if: github.event_name != 'workflow_dispatch' || !github.event.inputs.build_only
-      uses: docker/login-action@v1
+      uses: docker/login-action@v2
       with:
         registry: ghcr.io
         username: ${{ github.actor }}

.github/workflows/msys2.yml

@@ -42,13 +42,13 @@ jobs:
           mingw-w64-${{matrix.env}}-zeromq
           mingw-w64-${{matrix.env}}-boost
 
-    - uses: actions/checkout@v2
+    - uses: actions/checkout@v3
       with:
         submodules: true
       if: github.event_name != 'schedule'
 
     - name: Checkout develop branch
-      uses: actions/checkout@v2
+      uses: actions/checkout@v3
       with:
         submodules: true
         ref: develop

.github/workflows/static-analyzers.yml

@@ -19,7 +19,7 @@ jobs:
     container: helics/buildenv:cpplint
 
     steps:
-    - uses: actions/checkout@v2
+    - uses: actions/checkout@v3
     - name: Run cpplint
       run: cpplint --counting=detailed --recursive examples benchmarks src tests
 
@@ -29,7 +29,7 @@ jobs:
     container: helics/buildenv:clang-extra-tools
 
     steps:
-    - uses: actions/checkout@v2
+    - uses: actions/checkout@v3
     - name: Run clang-tidy on changed files
       shell: bash
       run: ./.github/actions/run-clang-tidy-pr.sh

.github/workflows/swig-interface-gen.yml

@@ -13,7 +13,7 @@ jobs:
     runs-on: ubuntu-latest
 
     steps:
-    - uses: actions/checkout@v2
+    - uses: actions/checkout@v3
       with:
         fetch-depth: 0
 

.pre-commit-config.yaml

@@ -8,15 +8,15 @@ repos:
       - id: dockerfile_lint
         args: [--rulefile, ./config/Docker/docker_rules.yml, --dockerfile]
   - repo: https://github.com/psf/black
-    rev: 22.10.0
+    rev: 22.12.0
     hooks:
       - id: black
         args: ["--line-length=100"]
   - repo: https://github.com/asottile/blacken-docs
-    rev: v1.12.1
+    rev: 1.13.0
     hooks:
       - id: blacken-docs
-        additional_dependencies: [black==21.10b0]
+        additional_dependencies: [black==22.12]
   - repo: https://github.com/Lucas-C/pre-commit-hooks
     rev: v1.3.1
     hooks:
@@ -52,7 +52,7 @@ repos:
         exclude: "mac.md"
       - id: script-must-have-extension
   - repo: https://github.com/shellcheck-py/shellcheck-py
-    rev: v0.8.0.4
+    rev: v0.9.0.2
     hooks:
       - id: shellcheck
         args: [-x]
@@ -74,7 +74,7 @@ repos:
             "--exclude-file=./config/spelling_ignorelines.txt",
           ]
   - repo: https://github.com/pre-commit/mirrors-clang-format
-    rev: v15.0.4
+    rev: v15.0.6
     hooks:
       - id: clang-format
         types:

CHANGELOG.md

@@ -9,7 +9,42 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
 A note on future revisions.
 Everything within a major version number should be code compatible (with the exception of experimental interfaces). The most notable example of an experimental interface is the support for multiple source inputs. The APIs to deal with this will change in future minor releases. Everything within a single minor release should be network compatible with other federates on the same minor release number. Compatibility across minor release numbers may be possible in some situations but we are not going to guarantee this as those components are subject to performance improvements and may need to be modified at some point. Patch releases will be limited to bug fixes and other improvements not impacting the public API or network compatibility. Check the [Public API](./docs/Public_API.md) for details on what is included and excluded from the public API and version stability.
 
-## [3.3.2][] ~ 2022-12-02
+## [3.4.0][] - 2023-01-19
+
+Major release including full dynamic federation capability and wildcard based matching for interface connections.
+
+### Fixed
+
+- Updated some out of date docker images to build automatically and fix some issues preventing the use of `-flto` option
+- Fixed a few more sporadically failing testing cases and a rare bug in the use of `wait_for_current_time` flag when entering executing mode
+
+### Changed
+
+- Test with Boost 1.81 and CMake 3.25
+- Update the units library
+- Update Spdlog library
+- Update to release 2.3 of CLI11
+- Split the application API tests in a set of 4 smaller tests to reduce test runtime and ease maintenance.
+- Update copyright to 2023
+
+### Added
+
+- Added the capability to link interfaces through a regex expression. `"REGEX: <regex>"` this format of string can be used in any place interface names can be used. This will not work on targets linked after initialization mode at present though this is expected to be relaxed in the future. Also if both sides are specified as strings at least one of them must be an actual name.
+- Added fully dynamic federation capability. Federates may be added dynamically if `--dynamic` flag is set on the root broker. Dynamic publications and endpoints are enabled by default but can be disabled.
+- Added automated workflow for generating release task checklist on version release
+- Added dependabot configuration for updating github-actions workflow dependencies
+- Added documentation on using encrypted communication in HELICS
+- Added support for using environment variables for enabling encryption instead of command line arguments for federates
+
+### Deprecated
+
+- Nearly all methods that use `Subscription` in the name have been deprecated in favor of an equivalent version for `Input` to reduce confusion of use of subscription objects (which are just Input objects); these are all 1:1 mappings
+  - `getSubscription(string target)` -> `getInputByTarget(string target)`
+  - `getSubscription(int index)` -> `getInput(int index)`
+  - `helicsSubscriptionGetTarget(inp)` -> `helicsInputGetTarget(inp)`
+  - `helicsFederateGetSubscription(fed, string target)`->`helicsFederateGetInputByTarget(fed,string target)`
+
+## [3.3.2][] - 2022-12-02
 
 Patch release to fix some testing issues and add some network probing capabilities and remote terminal connections.
 
@@ -30,7 +65,7 @@ Patch release to fix some testing issues and add some network probing capabiliti
 - Added a probe app for testing some networking situations and connections.
 - Added tests for the tcp federate under various conditions.
 
-## [3.3.1][] ~ 2022-10-21
+## [3.3.1][] - 2022-10-21
 
 Patch release to fix some issues with using HELICS as a subproject, and fix a few bugs related to time barriers, and a few other timing issues.
 
@@ -111,7 +146,7 @@ The release also includes several bug fixes related to timing and iteration.
 - ghc::filesystem, since all minimum compilers have support for std::filesystem available.
 - Removed Travis CI related configuration and documentation
 
-## [3.2.1][] - 2022-06-17
+## [3.2.1][] - 2022-06-16
 
 The Matlab interface to HELICS has been moved to [matHELICS](https://www.github.com/GMLC-TDC/matHELICS), the C++ interface now uses `std::string_view` in nearly all places, and the translator C API, and the dataBuffer API to the C interface were fleshed out a bit more and will be finalized in the 3.3 release. Subsequent releases will change the minimum compiler requirements to build HELICS to Visual Studio 2019, GCC 8.0, Clang 7, Xcode 11.0, and CMake 3.11.
 
@@ -344,3 +379,4 @@ HELICS 3.0 is a major update to HELICS. The major features that have been added
 [3.3.0]: https://github.com/GMLC-TDC/HELICS/releases/tag/v3.3.0
 [3.3.1]: https://github.com/GMLC-TDC/HELICS/releases/tag/v3.3.1
 [3.3.2]: https://github.com/GMLC-TDC/HELICS/releases/tag/v3.3.2
+[3.4.0]: https://github.com/GMLC-TDC/HELICS/releases/tag/v3.4.0

CMakeLists.txt

@@ -1,13 +1,13 @@
 # ~~~
-# Copyright (c) 2017-2022, Battelle Memorial Institute; Lawrence Livermore
+# Copyright (c) 2017-2023, Battelle Memorial Institute; Lawrence Livermore
 # National Security, LLC; Alliance for Sustainable Energy, LLC.
 # See the top-level NOTICE for additional details.
 # All rights reserved.
 #
 # SPDX-License-Identifier: BSD-3-Clause
 # ~~~
 
-cmake_minimum_required(VERSION 3.11...3.24)
+cmake_minimum_required(VERSION 3.11...3.25)
 
 # Install dependencies using vcpkg if VCPKG_ROOT is set and no CMake Toolchain file is given vcpkg
 # installation on a system doesn't set VCPKG_ROOT, so setting it should be like an opt-in for users
@@ -17,14 +17,14 @@ if(DEFINED ENV{VCPKG_ROOT} AND NOT DEFINED CMAKE_TOOLCHAIN_FILE AND NOT HELICS_D
     set(CMAKE_TOOLCHAIN_FILE "$ENV{VCPKG_ROOT}/scripts/buildsystems/vcpkg.cmake" CACHE STRING "")
 endif()
 
-project(HELICS VERSION 3.3.2)
+project(HELICS VERSION 3.4.0)
 
 # -----------------------------------------------------------------------------
 # HELICS Version number
 # -----------------------------------------------------------------------------
 set(HELICS_VERSION_BUILD)
 # use ISO date YYYY-MM-DD
-set(HELICS_DATE "2022-12-02")
+set(HELICS_DATE "2023-01-19")
 
 set(HELICS_VERSION_UNDERSCORE
     "${HELICS_VERSION_MAJOR}_${HELICS_VERSION_MINOR}_${HELICS_VERSION_PATCH}"
@@ -883,7 +883,7 @@ if(HELICS_ENABLE_PACKAGE_BUILD)
         "helics_app"
         "Helics app executable"
         "helics_recorder"
-        "Helics recorder"
+        "Helics Recorder"
         "helics_player"
         "Helics Player"
     )

LICENSE

@@ -1,6 +1,6 @@
 BSD 3-Clause License
 
-Copyright (c) 2017-2022, Battelle Memorial Institute; Lawrence Livermore National Security, LLC; Alliance for Sustainable Energy, LLC.
+Copyright (c) 2017-2023, Battelle Memorial Institute; Lawrence Livermore National Security, LLC; Alliance for Sustainable Energy, LLC.
 All rights reserved.
 
 Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: