v4.0.0

PrimAITE 4.0.0 Release Note

📰 Headlines 📰

• Major Release - Users are encouraged to also familiarise themselves with what's new in PrimAITE 3.0.0.
• Plugins - PrimAITE now supports external plugins to extend existing functionality, such as new node types, services, applications, agent types, observations, actions, and rewards.
• Improved config schema - The YAML config files have a new schema which simplifies the definition of agents by using named references in actions and observations instead of id references.
• Use Case 7 - Introduction of much larger UC7 network with example notebooks and config files.

✨ What's new ✨

• Use Case 7 is a new pre-defined scenario in PrimAITE with a more complex set of nodes that demonstrates some of the new capabilities of PrimAITE 4.
• Threat Actor Profiles are a new type of malicious scripted agent:
  • Extensible base class that allows defining a multi-stage kill chain with success conditions and network knowledge tracking.
  • TAP001 agent - agent that exfiltrates and corrupts data from a database.
  • TAP003 agent - agent that maliciously introduces ACL rules to a network to disrupt normal users.
• Users, Terminals, SSH, and Command&Control (introduced in PrimAITE 3.3).
• Determinism Support for determinism by setting and logging randomness seeds.
• Action Masking (introduced in 3.2.0).
• MARL support (introduced in 3.0.0).
• Logging was improved by adding the following:
  • detailed information about agent actions, decisions, and rewards.
  • the full state of the simulation after each environment step.
  • sys logs for each node.
  • pcap logs for each network interface.
• Domain Randomisation - the Gym environment can use different variations of the same scenario, alternating between them each episode by providing a folder of YAML files instead of a single file at initialisation.

👍 General Improvements 👍

• The organisation of the codebase has been improved by splitting long files into smaller ones.
• Agent logs can now show observation history and more detail about the reward.
• Some classes' .show() methods now show more useful or correct information (like agents, and networks).
• More example notebooks and introduction of how-to guides in the Sphinx docs.
• Ability to set scenario-wide default values for certain actions like scan, node power-off, node start-up, etc.
• It's now easier to build complex networks with the new, extensible NetworkNodeAdder class.
• NMAP application.
• The YAML config files:
  • support extended classes from plugins
  • information has been deduplicated - actions no longer rely on IDs, instead users can specify meaningful labels.
  • more objects have default values so there is less boilerplate.
  • the way agent settings are defined has been standardised.
  • more data validation was added to catch configuration errors earlier.
  • PrimAITE 3 to PrimAITE 4 YAML migration guide.
• Observations can be configured to not require a scan action to show the true health state of software or files.

🐛 Bug Fixes 🐛

•   DNS client no longer fails to check its cache if a DNS server address is missing.
•   DNS client now correctly inherits the node's DNS address configuration setting.
•   ACL observations now include the ACL at index 0.
•   SoftwareManager show method correctly displays all the software associated with a port whether the software is listening or not.

🚦 Tests

This release was checked against specified unit/integration test suite across Windows, Linux and MacOS with Python 3.9-3.11.

⌛ Deprecated ⌛

PrimAITE can no longer be used with Ray versions < 2.32.

⚠️ Breaking Changes ⚠️

This is a major release so many classes and methods have changed as well as the config files. See the documentation for further details of API changes and YAML Migration Guide for config updates.

🔍 Known Issues 🔍

• Action Masking - There is a known issue with the API stack for Ray RLlib affecting the saving of RL policies. A suggested workaround can be found on this issue ticket.
• Agent actions - Agents action spaces must use the gymnasium.spaces.Discrete shape, and a list of actions must be provided using the action_map config option for each agent. Other action spaces and more flexible action space definition will be part of a future release.
• Determinism - Ray RLLib MultiAgent environments are not reproducing identical training runs even after setting random seeds in PrimAITE. This might be caused by the spawning of multiple threads for the RL algorithm.
• Random Number Generators (RNG) with different devices - When using StableBaselines3, different results will be generated for the same seed if trained on a CPU vs a GPU. For example: WSL will perform training on the CPU but Windows will attempt to train on a graphics card if possible. In the notebook Training-an-SB3-Agent add the following argument to the PPO command (cells 6 and 9): device=cpu,  to force training on the CPU rather than GPU. NB: This can impact training times. This issue has not been observed with Single Agent Ray RLLib training.
• NMNE - when using NMNE, both config parameters include_nmne and  capture_nmne must be set to the same Boolean value.

⬆️ Upgrade Instructions ⬆️

From GitHub

pip install git+https://github.com/Autonomous-Resilient-Cyber-Defence/primaite@v4.0.0#egg=primaite

From wheel

pip install path/to/primaite-4.0.0-wheel.whl[rl] --upgrade

As a repo

git clone https://github.com/Autonomous-Resilient-Cyber-Defence/primaite
cd PrimAITE/
git fetch
git checkout tags/v4.0.0
pip install -e .[rl] --upgrade

👥 Contributors 👥

@pufferfish-seaweed
@CharlieC-QQ
@MethodsGRS
@jamesshort1
@njtodd
@marek-methods
@ChrisMcCarthyDev
@CGenes-Methods

📚 References 📚

• PrimAITE 4.0.0 is compatible with Common Action and Observation Space (CAOS) v 0.10.3, a copy of which has been provided with this release.

The Common Action / Observation Space (CAOS) defines a common language for agents to adopt when interacting with PrimAITE and potential other environments. It consists of an evolving definition of Blue, Green and Red Agent Action and Observation Spaces compatible with the MITRE frameworks and is employed to define an agent-environment interface. CAOS was born out of a necessity for environments to speak ‘one language’ with external agents; this is essential to support the transfer of agents that have undergone training in a simulator, and then require evaluation or demonstration in a higher fidelity environment.

v3.3.0

PrimAITE 3.3.0 Release Note

📰 Headlines 📰

• Terminal software - A new service which allows the execution of arbitrary commands on nodes by leveraging the existing request system. Terminals on different nodes communicate via the SSH protocol allowing remote command execution.
• Local accounts - Nodes now support local user accounts with regular or elevated privileges. This integrates with the terminal to simulate scenarios involving stolen credentials.
• Command and control - A set of malicious services which can infect computers and execute remotely, including sending SSH traffic masked as other protocols.

✨ What else is New ✨

• Determinism - PrimAITE now supports setting a random seed which ensures reproducible behaviour. This has been tested with seeded stable-baselines3 and Ray RLLib agents which produces identical training runs.
• Login and logout actions - New agent actions were added to allow agents to log in and out of nodes.
• User account observations - Agents can now observe how many users are logged in to each node.
• Terminal actions - Agents can send terminal commands as actions.
• Reward component stickiness - Green and Blue agents can optionally return reward data per timestep regardless of whether the corresponding agent executed an action.

🚀 Improvements 🚀

• Multi-port listening - Software can now listen on multiple network ports.
• Enhanced agent history - Agent history includes more information about how the reward was calculated.
• Richer response data - Response data was added to some request types which previously didn't return additional success data.

🐛 Bug Fixes 🐛

• Folder observations - no longer necessary to scan to get true health state.
• SoftwareManager update
  • install and uninstall methods replace functionality previous handled by equivalent methods in Node class.
  • receive_payload_from_session_manager sends copy of data to any software listening on the destination port of the Frame.

🚦 Tests

• Tested against the a beta release checklist to ensure the software does not crash and produces expected results.

⌛ Deprecated ⌛

No features were deprecated in this update.

🔍 Known Issues 🔍

• Agent actions - Agents action spaces must use the gymnasium.spaces.Discrete shape, and a list of actions must be provided using the action_map config option for each agent. Other action spaces and more flexible action space definition will be part of a future release.
• Ray incompatibility with Python 3.8 - Recent versions of Ray fail to install on Python 3.8. Users who are using Python 3.8 cannot use the [rl] optional dependency flag when installing PrimAITE; they must manually install their desired version of Ray.
  i.e. Instead of running pip install path/to/primaite/wheel.whl[rl], users should run pip install path/to/primaite/wheel.rl and then pip install ray[rllib]
  This issue is not present when installing PrimAITE on Python 3.9 and later.
• Determinism - RayMultiAgent RL appears to be immune to random number generator seed setting as it's currently implemented and will continue to generate non-determinsistic output even if a seed is set.
  When using StableBaselines 3, different results will be generated for the same seed if trained on a CPU vs a GPU. For example: WSL will perform training on the CPU but Windows will attempt to train on a graphics card if possible. In the notebook Training-an-SB3-Agent add the following argument to the PPO command (cells 6 and 9): device=cpu, to force training on the CPU rather than GPU. NB: This can impact training times.
  This issue has not been observed with Single Agent Ray RLLib notebooks.
• NMNE - when using NMNE both config parameters include_nmne and capture_nmne must be set to the same Boolean value.

⬆️ Upgrade Instructions ⬆️

From wheel

pip install path/to/primaite-3.3.0-wheel.whl[rl] --upgrade

As a repo

git fetch
git checkout tags/v3.3.0
pip install -e .[rl] --upgrade

👥 Contributors 👥

@pufferfish-seaweed
@CharlieC-QQ
@czar-ec-envitia
@ChrisMcCarthyDev
@MethodsGRS
@jamesshort1
@njtodd
@marek-methods

📚 References 📚

PrimAITE 3.3.0 supports CAOS 0.10 but the new CAOS features are optional so it is still possible to use configs designed around CAOS 0.9

v3.2.0

PrimAITE 3.2.0 Release Note

📰 Headlines 📰

• Link bandwidth limiting was reenabled. Links now correctly respect their bandwidth and refuse to transmit data once the limit is reached in that timestep. Frames arriving at a full link are dropped. This also applies to the airspace which has a single bandwidth shared between all devices using the airspace for each given frequency.
• Action masking was added to all environment types. PrimAITE can provide a boolean vector corresponding to RL agents' action lists to show which ones are valid at the current timestep. This can greatly increase training speed for RL algorithms that support action masks.
• Agent logging was improved. Each agent now produces a separate log file which contains information about the internal logic of the agent.

✨ What else is New ✨

• Configure actions - DatabaseClient, RansomwareScript, and DoSBot have new CONFIGURE actions associated with them that allow setting application-specific attributes. These can be used by red agents to set the attack target after successful network discovery.
• Action penalty - There is a new reward component that can be configured which applies a positive/negative reward for taking actions (other than the DONOTHING action).

🚀 Improvements 🚀

• NODE_APPLICATION_INSTALL action was improved to be more flexible. It can now be used to install any application. It no longer requires an ip_address parameter, ip configuration is handled by new CONFIGURE actions.
• Action validation was improved in the simulation. Additional checks verify if an action is currently possible and a more helpful failure message is produced if the criteria for resolving an action were not met.
• Database fixing - The database no longer replies to queries while it is being fixed, preventing agents from spamming the repair action to maximise reward.
• Fixing duration - Fixing duration is now configurable per-service.

🐛 Bug Fixes 🐛

• Incorrect default action spaces - NODE_APPLICATION_EXECUTE actions were removed from blue agents in the default configs.

🚦 Tests

This release has been tested against the full PrimAITE release process, including system-level testing, user testing, and design review.

⚠️ Breaking Changes ⚠️

• NODE_APPLICATION_INSTALL actions no longer accept an ip_address parameter. If your custom configs use this action, you need to remove this parameter. You might need to add CONFIGURE actions to your agents that previously relied on this.

🔍 Known Issues 🔍

• Agent actions - Agents action spaces must use the gymnasium.spaces.Discrete shape, and a list of actions must be provided using the action_map config option for each agent. Other action spaces and more flexible action space definition will be part of a future release.
• Ray incompatibility with Python 3.8 - Recent versions of Ray fail to install on Python 3.8. Users who are using Python 3.8 cannot use the [rl] optional dependency flag when installing PrimAITE; they must manually install their desired version of Ray.
  i.e. Instead of running pip install path/to/primaite/wheel.whl[rl], users should run pip install path/to/primaite/wheel.rl and then pip install ray[rllib]
  This issue is not present when installing PrimAITE on Python 3.9 and later.

⬆️ Upgrade Instructions ⬆️

From wheel

pip install path/to/primaite-3.2.0-wheel.whl[rl] --upgrade

As a repo

git fetch
git checkout tags/v3.2.0
pip install -e .[rl] --upgrade

v3.1.0

PrimAITE 3.1.0

📰 Headlines 📰

• Ping scan and port scan - PrimAITE includes a new application called NMAP, which allows agents to perform a ping scan or a port scan on the network. This is accompanied by ping / port scan actions which return data back to the agent to dynamically select target nodes.
• CAOS v0.9 - PrimAITE 3.1 is compliant with Common Action and Observation Space (CAOS) v0.9. This is a minor change that introduces a new traffic observation on network interfaces. The CAOS v0.9 specification is attached.

✨ What's New ✨

• Network interface traffic observation - It is possible for the observation space to report network traffic amounts on network interfaces, broken down by protocol, port, and whether the traffic is inbound or outbound. To include this observation component, the network interface observation, or any of its parents must include the key monitored_traffic. For example:

  observation_space:
    type: NODES
    options:
      # ... other options
      monitored_traffic:
        tcp:
          - DNS
          - HTTP
        icmp:
          - NONE
      # ... other options

🚀 Improvements 🚀

• Agent reward logging - All past agent rewards are now reported within the agent history log file.

• Notebooks - Documentation notebooks have been updated to reflect the new changes.

🐛 Bug Fixes 🐛

• Database client uninstall action - Fixed an issue where the program would enter an infinite loop when using the application uninstall action on the database client in certain circumstances.

⚠️ Breaking Changes ⚠️

• target_router_nodename config key was renamed to target_router. Users should find-and-replace any instances of target_router_nodename in the action map part of their config files.

CAOS Compatibility

CAOS v0.9 has been developed alongside this release of PrimAITE. The following changes were made from CAOS v0.8 to v0.9:

• Corrected BAS-62 -> BAS-70 (APPLICATION:FIX). Had the wrong ID
• New BOS-120 and BOS-121 which present inbound and outbound traffic levels per protocol / port on each NIC

CAOS Version	Supports Use Case	Compatible Releases
0.7	UC2	v3b7
0.8	UC2	v3b8, v3b9, v3.0.0
0.9	UC2	v3.1.0, v3.2.0

v3.0.0

PrimAITE v3.0.0

PrimAITE was rebuilt from the ground up. This release note is aimed at users of PrimAITE 2 to highlight the key changes and new concepts. However, both new and returning users are encouraged to read the docs and example notebooks because of the magnitude of changes since the last release.

✨ What's New

High-level software architecture

• PrimAITE now consists of a standalone network simulation and a game layer which sits on top of the simulation and turns it into an environment for training RL agents.
• Request and State - The game layer receives simulation data via the state interface, and it sends agents' actions via the request interface. Refer to the docs for a full description.
• Common Action and Observation Space (CAOS) - PrimAITE 3.0.0 adheres to CAOS v0.8 specification.
• Multi-agents - Red, green, and blue agents all inherit from the same interface, allowing arbitrarily many scripted and RL agents.
• Pydantic - PrimAITE uses Pydantic throughout for type safety

Simulation

• Fidelity - PrimAITE has the ability to model layers 1-5 of the OSI networking model.
• Packets - Network traffic consists of packets with realistic headers.
• Routers, switches, and firewalls contain internal logic such as ARP tables, route tables, and ACLs.
• Software - Software on nodes can generate network traffic, respond to actions, and interact with other software and files.
• File system - Host nodes now have a file system with folders and files.

Game Layer

• Agents - The game layer now contains red and green agents which displace the previous notions of IERs and PoL. Agents are classes with custom logic for selection CAOS actions in response to observations.
• Actions - Agent actions are interpreted as a simulation request. Agents receive a request response with a success status and data generated by the action.
• Observations - Blue agent observations are configured hierarchically and they are based on the state interface of the simulation. Agents can have a view over everything or they can be specialised to see certain parts of the network.
• Rewards - Red, green, and blue agents all support rewards, which can take into account the simulation state, and the agent's past actions. Agents' rewards can also contribute to other agents' rewards.

Configuration

• Schema - There is a new schema for configuring PrimAITE scenarios. This is explained in the full documentation.
• Episode schedule - Curriculum learning and enhanced domain randomisation is possible by creating variations around a base config that change between episodes. This is achieved by creating a folder containing a base config with placeholders and files with options for populating those placeholders. The user specifies a schedule listing which placeholders to use in each episode. This allows for training defensive agents against a variety of red agents, green agent behaviours, or even different network layouts.

Running PrimAITE

• Notebooks - New Jupyter notebooks have been provided to demonstrate examples of agent training, saving, loading, and evaluation using SB3 and Ray RLLib. These displace the Session module from PrimAITE 2 - PrimAITE 3 focuses purely on providing an RL environment, not any training / evaluation infrastructure.
• Support - As a pure Python package, PrimAITE supports Windows, Linux, and MacOS.

🐞 Bug Fixes

The bugs present in PrimAITE 2 are no longer present as a result of the software rewrite. This includes:
Transaction CSV incorrect headers - PrimAITE 3 logging no longer uses a CSV file for transactions, however agent action history and step-wise environment state can be logged which provides the same information.

PrimAITE not installing on Mac Sonoma 14.2.1 - MacOS installation success is assured as part of the CI builds that are part of PrimAITE development cadence.

ACL action filtering incorrect with implicit ALLOW - the ACL system has been rewritten to work with the new node system. This has been fixed.

Traffic always blocked - This bug was caused by inconsistent data type handling in PrimAITE 2. PrimAITE 3 is using Pydantic to avoid issues like this. This has been fixed.

Documentation not rendering properly - The documentation is not currently hosted online but it can be built locally.

🔨 Breaking Changes

Code and configuration files written for PrimAITE 2 are not compatible with PrimAITE 3.

♻️ Refactoring

Since the software was rewritten, there are too many refactoring items to list here, but the software structure is described in the documentation.

🚦 Tests

📚 Docs

Rendered sphinx docs are attached to the PrimAITE 3.0.0 GitHub release page. Users can also build the docs manually after installing PrimAITE by running the following:

cd docs
make html

• User guide and developer docs - The user guide, developer docs, and API reference have been updated to reflect the new codebase.
• Jupyter notebooks - Several notebooks are provided with example usage, and explanations of new features. Upon installation, these are copied to the user data directory (~/primaite/3.0.0/notebooks) for running interactively. Static versions of the notebooks are also available within the sphinx documentation.

⚡️ Performance Notes

Users should note that the heavy demands made on CPU and RAM by some PrimAITE activities, particularly when a complex scenario is modelled, may result in other applications running slower than they would on an otherwise unloaded system.

⚠️ Known Issues

• Bandwidth - Link bandwidth has been disabled; links are able to transmit an unlimited amount of data each step. Therefore, it is not possible to model certain types of denial of service attacks. The link bandwidth is only used for observations to report percent utilisation, but if the load exceeds bandwidth, they are treated as 100% loading.
• Agent actions - Agents action spaces must use the gymnasium.spaces.Discrete shape, and a list of actions must be provided using the action_map config option for each agent. Other action spaces and more flexible action space definition will be part of a future release.
• Open file limit exceeded - On certain platforms, loggers might open too many file descriptors. This can be circumvented by increasing the open file limit (such as by using ulimit -n 9999 on linux), or by disabling PCAP logs via the config.

💫 Getting started

Installing from wheel

• On Linux, create your virtual environment and install PrimAITE:

  python -m venv path-to-your/venv
  source path-to-your/venv/bin/activate
  pip install path-to-your/primaite-3.0.0-wheel.whl[rl]
  primaite setup
  

• On Windows, create your virtual environment and install PrimAITE:

  python -m venv path-to-your\venv
  path-to-your\venv\Scripts\Activate.ps1
  pip install path-to-your\primaite-3.0.0-wheel.whl[rl]
  primaite setup

Installing from repository

• On Linux, clone PrimAITE, and install:

  git clone https://github.com/Autonomous-Resilient-Cyber-Defence/PrimAITE.git --branch v3.0.0
  cd PrimAITE
  python -m venv venv
  source venv/bin/activate
  pip install -e .[rl]
  primaite setup

• On Windows, clone PrimAITE, and install

  git clone https://github.com/Autonomous-Resilient-Cyber-Defence/PrimAITE.git --branch v3.0.0
  cd PrimAITE
  python -m venv venv
  venv\bin\Activate.ps1
  pip install -e .[rl]
  primaite setup

🛠 Engineering Notes

• After installing PrimAITE, if you wish to run in dev mode, run the following commands:

  pip install -e .[rl,dev]
  primaite setup
  primaite mode dev

  This overrides logging options to output logs to the project directory instead of the user data directory.

Contributors

@pufferfish-seaweed
@CharlieC-QQ
@czar-ec-envitia
@ChrisMcCarthyDev
@njtodd
@marek-methods

v2.0.0

PrimAITE v2.0.0

✨ What's New

Command Line Interface

PrimAITE now comes with a CLI (built with Typer) that serves as the main entry point for those using PrimAITE out of the box.

To run the default PrimAITE Session out of the box, run:

primaite session

Application Directories

To enable PrimAITE to be used as an installed Python package, and to be used as is out-of-the-box without reliance on the repository, a collection of application directories has been created. These back-end/hidden and user-facing directories are used to store things like application log files, users' config files, users' Jupyter Notebooks, PrimAITE session outputs etc. The user needs to call primaite setup after doing pip install to perform the directory setup. The directories are structured as follows:

Windows

~/
├─ AppData/
│  ├─ primaite/
│  │  ├─ 2.0.0/
│  │  │  ├─ config/
│  │  │  ├─ logs/
│  │  │  │  ├─ primaite.log
├─ primaite/
│  ├─ 2.0.0/
│  │  ├─ config/
│  │  │  ├─ example_config/
│  │  │  │  ├─ lay_down/
│  │  │  │  ├─ training/
│  │  ├─ notebooks/
│  │  │  │  ├─ primaite_demo_notebooks/
│  │  ├─ sessions/

Linux

~/
├─ .cache/
│  ├─ primaite/
│  │  ├─ 2.0.0/
│  │  │  ├─ logs/
│  │  │  │  ├─ primaite.log
├─ .config/
│  ├─ primaite/
│  │  ├─ 2.0.0/
├─ .local/
│  ├─ share/
│  │  ├─ primaite/
│  │  │  ├─ 2.0.0/
├─ primaite/
│  ├─ 2.0.0/
│  │  ├─ config/
│  │  │  ├─ example_config/
│  │  │  │  ├─ lay_down/
│  │  │  │  ├─ training/
│  │  ├─ notebooks/
│  │  │  │  ├─ primaite_demo_notebooks/
│  │  ├─ sessions/

MacOS

~/
├─ Library/
│  ├─ Application Support/
│  │  ├─ Logs/
│  │  │  ├─ primaite/
│  │  │  │  ├─ 2.0.0/
│  │  │  │  │  ├─ log/
│  │  │  │  │  │  ├─ primaite.log
│  │  ├─ Preferences/
│  │  │  ├─ primaite/
│  │  │  │  ├─ 2.0.0/
│  │  ├─ primaite/
│  │  │  ├─ 2.0.0/
├─ primaite/
│  ├─ 2.0.0/
│  │  ├─ config/
│  │  │  ├─ example_config/
│  │  │  │  ├─ lay_down/
│  │  │  │  ├─ training/
│  │  ├─ notebooks/
│  │  │  │  ├─ primaite_demo_notebooks/
│  │  ├─ sessions/

Support for Ray Rllib

PrimAITE now supports the training of PPO and A2C agents using both Stable Baselines3 and Ray RLlib. The RL framework and agent algorithm to be used for training is determined by the agent_framework and agent_identifier configurable items in the training config file. If agent_framework=RLLIB, the backend Ray RLlib RL framework can be configured to use either Tensorflow, Tensorflow 2.x, or PyTorch using the deep_learning_framework.

# Sets which agent algorithm framework will be used.
# Options are:
# "SB3" (Stable Baselines3)
# "RLLIB" (Ray RLlib)
# "CUSTOM" (Custom Agent)
agent_framework: RLLIB

# Sets which deep learning framework will be used (by RLlib ONLY).
# Default is TF (Tensorflow).
# Options are:
# "TF" (Tensorflow)
# TF2 (Tensorflow 2.X)
# TORCH (PyTorch)
deep_learning_framework: TF2

# Sets which Agent class will be used.
# Options are:
# "A2C" (Advantage Actor-Critic coupled with either SB3 or RLLIB agent_framework)
# "PPO" (Proximal Policy Optimization coupled with either SB3 or RLLIB agent_framework)
# "HARDCODED" (The HardCoded agents coupled with an ACL or NODE action_type)
# "DO_NOTHING" (The DoNothing agents coupled with an ACL or NODE action_type)
# "RANDOM" (primaite.agents.simple.RandomAgent)
# "DUMMY" (primaite.agents.simple.DummyAgent)
agent_identifier: PPO

Random red agent

A random red agent has been provided to train the blue agent against. The random red agent will choose a random number of nodes to attack, as well as randomly choosing the actions to perform on the environment.

# Sets whether Red Agent POL and IER are randomised.
# Default is False.
# Options are:
# True
# False
random_red_agent: False

Repeatability of sessions

A seed can now be provided in the training configuration file. The seed will be used across PrimAITE so that a repeatable run is achievable. The seed needs to be an integer value and by default is set to null.

The ability to set PrimAITE to use deterministic or stochastic evaluation is also added.

# The (integer) seed to be used in random number generation
# Default is None (null)
seed: null

# Set whether the agent evaluation will be deterministic instead of stochastic
# Default is False (stochastic).
# Options are:
# True
# False
deterministic: False

Session loading

PrimAITE can now load previously run sessions for SB3 Agents (SB3 agents only, see Known Issues. This can be done via:

CLI

primaite session --load "<PREVIOUS_SESSION_DIRECTORY>"

Python

from primaite.main import run

run(session_path=<PREVIOUS_SESSION_DIRECTORY>)

The output for the loaded session will be in the target directory i.e. the PREVIOUS_SESSION_DIRECTORY. While most of the outputs won't be overwritten, the agent zip file will be overwritten.

Agent Session Classes

An AgentSessionABC class with SB3Agent and RLlib subclasses and a HardCodedAgentSessionABC class with various hard-coded agent subclasses have been created. The Agent Session classes act as a wrapper around various RL and hard-coded agents. They help to standardise how agents are trained in PrimAITE using a common interface. They also provide a suite of standardised session outputs.

Standardised Session Output

When a session is run, a session output sub-directory is created in the user's app sessions directory
(~/primaite/sessions). The sub-directory is formatted as such: ~/primaite/sessions/<yyyy-mm-dd>/<yyyy-mm-dd>_<hh-mm-dd>/. This session directory is populated with four types of outputs:

• Session Metadata
• Results
• Diagrams
• Saved agents (training checkpoints and a final trained agent)

Example Session Directory Structure

~/
└── primaite/
    └── 2.0.0/
        └── sessions/
            └── 2023-07-18/
                └── 2023-07-18_11-06-04/
                    ├── evaluation/
                    │   ├── all_transactions_2023-07-18_11-06-04.csv
                    │   ├── average_reward_per_episode_2023-07-18_11-06-04.csv
                    │   └── average_reward_per_episode_2023-07-18_11-06-04.png
                    ├── learning/
                    │   ├── all_transactions_2023-07-18_11-06-04.csv
                    │   ├── average_reward_per_episode_2023-07-18_11-06-04.csv
                    │   ├── average_reward_per_episode_2023-07-18_11-06-04.png
                    │   ├── checkpoints/
                    │   │   └── sb3ppo_5.zip
                    │   ├── SB3_PPO.zip
                    │   └── tensorboard_logs/
                    │       ├── PPO_1/
                    │       │   └── events.out.tfevents.1689674765.METD-9PMRFB3.42960.0
                    │       ├── PPO_2/
                    │       │   └── events.out.tfevents.1689674766.METD-9PMRFB3.42960.1
                    │       ├── PPO_3/
                    │       │   └── events.out.tfevents.1689674766.METD-9PMRFB3.42960.2
                    │       ├── PPO_4/
                    │       │   └── events.out.tfevents.1689674767.METD-9PMRFB3.42960.3
                    │       └── PPO_5/
                    │           └── events.out.tfevents.1689674767.METD-9PMRFB3.42960.4
                    ├── network_2023-07-18_11-06-04.png
                    └── session_metadata.json

PrimaiteSession Class

The PrimaiteSession class acts as a single wrapper around the Agent Session classes. It is both an entry point and a broker for the individual Agent Session classes.

Action Space

Discrete Action Space

The NODE and ACL action spaces have been changed from multi-discrete to discrete action spaces.

NODE and ACL action spaces are both dictionaries where a single number reflects an entire action an agent can take.

The below code block is an example of a dictionary entry for the NODE action space:

{
    1: [1, 1, 1, 0], 
    
}

The below code block is an example of a dictionary entry for the ACL action space:

{
    1: [1, 0, 1, 2, 1, 0, 3],
}

Combined Action Spaces

A new ANY action space option has been introduced. This allows the agent to do both NODE actions and ACL actions in the same episode (e.g., scan a node in Step 1 and create an ACL rule in Step 2).

The below code block is an example of a dictionary entry for the ANY action space:

{
    0: [1, 0, 0, 0], 
    1: [1, 1, 1, 0], 
    2: [1, 0, 1, 2, 1, 0, 3]
}

is_valid_acl_action_extra, is_valid_node_action and is_valid_acl_action in the primaite.agents.utils module help to slim down the dictionary to contain the relevant actions only.

For example, a node action to do PATCHING on a node's hardware state CANNOT happen so it is not added to the dictionary.

transform_action_node_readable and transform_action_acl_readable in the primaite.agents.utils module converts the enumerated node action into a more readable form. The readable form is used by functions such as is_valid_node_action to determine if the action is valid or not.

An example using transform_action_node_readable:

{ 
    # Converts node action into readable form
    [1, 3, 1, 0] -> [1, 'SERVICE', 'PATCHING', 0]
}

ACL Action Space

Previously, the ACL action space was made up of 6 items: [Decision, Permission, Source, Dest, Protocol, Port].

Now the agent can specifically choose where to place the ACL in the `A...