All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog and this project adheres to Semantic Versioning.
- Embed data file to make packaging easier.
0.10.3 - 2024-05-19
- Updates in the Nix bindings, improving stability.
0.10.2 - 2024-05-03
- Add
nixSettings
configuration item, to more easily configure Nix settings such assubstituters
, overriding the system Nix settings. - Cachix 1.7.2 support
- Remove most uses of
c_str()
, of which at least one exhibited undefined behavior. nix-darwin
module now readssystem
correctly from the configuration.
0.10.1 - 2024-02-12
-
More work is performed concurrently during evaluation, including binary cache lookups and (more) build dispatch. This results in a speedup.
-
Dependencies of build dependencies are not scheduled eagerly anymore. This reduces the scope of all jobs that are evaluated by agents since this release, resulting in a speedup. This resolves a noticable slowdown when first evaluating significant Nixpkgs updates when its
staging
branch is merged.Strictly speaking, a job success no longer guarantees that absolutely everything (all the way up to the bootstrap binaries) is realisable on your agents. This property is generally not your responsibility, and enforcing it had the effect of excluding less reproducible platforms such as darwin. Instead, a weaker property is provided: your derivations are realisable, as well as the immediate build dependencies. "Your derivations" is defined as those whose outputs are not already cached.
CI setups based on the Nix command line interface (almost all CIs) also behave this way.
-
The recommended configuration format is now JSON, preferably generated using a configuration manager such as NixOS or nix-darwin. TOML is still supported, but does not support
null
in labels, and due to library limitations, it requires that intermediate tables be specified. See the config file documentation. -
services.hercules-ci-agent
is now an alias forservices.hercules-ci-agents.""
, which still provides the same behavior as the old module. -
Hardening flags have been applied to the NixOS module.
-
The effect sandbox now use the
crun
container runtime instead ofrunc
. -
Attribute sets containing a
_type
attribute are not scanned for derivations inherculesCI.<...>.outputs
. This prevents accidental scanning of large or failing attribute trees, such as NixOS configurations.nixosConfigurations
in Flakes are still built as usual, as they are not (verbatim) in theherculesCI.<...>.outputs
attributes.
-
Effect mounts. Specify
effectMountables
in the agent configuration, deploy, and mount them into an effect. This can be used for instance to expose the host's/etc/hosts
, or hardware devices such as GPUs. Access is controlled by the agent configuration. -
New configuration option
remotePlatformsWithSameFeatures
, allowing a remote build to be used before more elaborate remote builder support is implemented. The recommended method for running a cluster is still to installhercules-ci-agent
on each machine, as that is more efficient and accurate. -
Agent labels can now be
null
, when using the JSON configuration format.
-
Low level crash details are now reported in the log as expected.
-
An interaction between the Nix GC and threads has been fixed, solving such a crash.
0.9.12 - 2023-06-28
-
Nix 2.16 support
-
Nix 2.15 support
- Do not
chdir
the build worker. This functionality of theprocess
package appears unreliable, but is not needed.
0.9.11 - 2023-03-06
-
The
nix-darwin
module uses new user id and group id numbers, to match the upstreamnix-darwin
module. The main benefit of the upstream change is that the agent user will not appear as a normal (human) user on the system.To migrate, run:
# if you have deployed an older version of the nix-darwin PR before, # the following delete commands will prevents this error: # creating group _hercules-ci-agent... # <main> attribute status: eDSRecordAlreadyExists # <dscl_cmd> DS Error: -14135 (eDSRecordAlreadyExists) sudo dscl . -delete '/Groups/_hercules-ci-agent' sudo dscl . -delete '/Users/_hercules-ci-agent' # update the flake inputs / expressions / channels # and deploy the new version, e.g: sudo darwin-rebuild switch sudo chown -R _hercules-ci-agent:_hercules-ci-agent /var/lib/hercules-ci-agent
-
Non-builder build errors such as output cycles are now reported in the build log.
-
ciSystems
is now taken into account by the defaultonPush.default
job when theherculesCI
attribute of a flake is a function. -
darwinConfigurations
is now filtered in accordance withciSystems
.
-
Nix verbosity can now be specified in the config file under the new attribute
nixVerbosity
. -
Cachix 1.3 support
-
Nix 2.14 support
-
Nix 2.13 support
0.9.10 - 2022-12-29
-
Detect stack overflows correctly in Nix evaluation
-
Retry errors from Nix-native (non-cachix) caches
- Cachix 1.1 compatibility
- Unwrap some error messages for readability
0.9.9 - 2022-12-02
-
The flake
templates
check now allows filtered sources. -
Tweaks to diagnostic messages
0.9.8 - 2022-11-15
-
herculesCI.onSchedule
jobs, which a created at set times. These can be used to automate work that doesn't start with a code change, such as automatic updates or impure periodic deployments. -
Secret types and the
GitToken
type, to be provided by the Hercules CI GitHub App. As of writing this requires a permission approval in the GitHub UI. This has to be initiated on the Hercules CI side. More on that very soon. -
More repository metadata in the
herculesCI
attribute; see evaluation docs.
-
Work around excessive stack use by libstdc++ regex issue The new limit of 256 MiB stack allows larger string inputs to be used. This bug was triggered by purs-nix and possibly other Nix expression libraries that parse substantial files.
-
Lift the agent-side restriction on the number of derivations in a job.
0.9.7 - 2022-07-21
- Nix 2.10 support
-nix_2_7
variants. Nixpkgs has stopped offering older versions of Nix. If you need an older version, you could inject an older Nixpkgs into this flake usinghercules-ci-agent.inputs.nixpkgs.follows = ...
.
0.9.6 - 2022-06-21
- Nix 2.9 support
- NixOS module: keep main process running when worker triggers system OOM killer.
- Missing file in hackage sdist
0.9.5 - 2022-05-17
- Concurrent IFD, reducing evaluation wall clock time
- Nix 2.8.0 support
- Improved log contexts
- Workaround for cachix#406 (add
login
tonetrc
) - A crash in
inline-c-cpp
exception handling (inline-c-cpp
update) - Towards the error "Could not push logs within 10 minutes after completion"
- Add a timeout to prevent hang in case of a stuck handshake
- Enforce log limit on client side as well in case of excessive log spam and an upload bottleneck
hercules-ci-agent-nix_2_5
variant: upgrade to plainhercules-ci-agent
(2.8.0) or_nix_2_7
.
Bad release. See 0.9.5 for changes.
0.9.3 - 2022-04-08
- The evaluator now caches build statuses and ignores redundant rebuild requests, giving a significant performance boost to IFD-heavy jobs.
- The branch name in flake-based jobs is now handled correctly when special characters are present.
0.9.2 - 2022-03-30
- Separate traces in the dashboard (as in
--show-trace
)
- Effects:
error: cannot open connection to remote store 'daemon': error: reading from file: Connection reset by peer
0.9.1 - 2022-03-18
- The built-in flake support now has the
sourceInfo
attributes.
- Flakes are checked out by Nix rather than custom local checkout.
0.9.0 - 2022-03-15
This release comes with an Upgrade Guide! ✨
-
Flakes support!
Instead of needing a
ci.nix
, the agent will pick upflake.nix
and look for theherculesCI
attribute in the flake.Only the
outputs.effects
sub-attributes may define effects, making attacks on secrets harder to conceal. -
Multiple jobs per commit
-
Jobs that run with the latest successful dependency build
-
Conditions on secrets, disallowing access to secrets except when the conditions are met. This enforces the four eyes principle when branch protection is set up to match the secrets' conditions. A missing
condition
field does not give a great error message for security reasons, so follow the upgrade guide. -
Hardening against rogue contributors. Trivial attacks trying to read system paths or secrets are no longer possible. Similar to typical CIs, secrets can be stolen under specific circumstances: either a misconfiguration of branch protection or by approval of a second maintainer. Note that issue was already largely addressed by only processing contributions from GitHub users with write access to the repository, which also still applies.
-
Built-in support for fetching private repositories and tarballs.
-
File lookup order has changed, to support flakes.
ci.nix
ornix/ci.nix
still take top priority, followed byflake.nix
, followed bydefault.nix
. -
Installed private repositories can now be read by a collaborator. If you need to enforce confidentiality across repositories, contact us and use a personal access token with appropriate permissions in the meanwhile.
- When the root of a
ci.nix
is a list, an error message is returned.
- Nix 2.3 support
0.8.7 - 2022-03-09
- Nix 2.7 support
- Haskell
aeson
2.0 support
0.8.6 - 2022-03-07
- Build with newer Nix versions 2.5, 2.6
- Improved conditional code support with
cabal-pkg-config-version-hook
- The flake now has
.nixosModules.multi-agent-service
allowing multiple agents to run on the same system. An instance with default settings can be enabled withservices.hercules-ci-agents."some-name" = {}
. User name and file paths are like the regular module, except replacinghercules-ci-agent
byhci-${name}
if the chosenname
is not""
.services.hercules-ci-agents."" = {}
is equivalent toservices.hercules-ci-agent.enable = true
.
-
Fix mounting
/etc/resolv.conf
, work around runc#1523. Fixed by @Mic92 in #357 -
An issue where a Nix evaluator crash could lead to builds being triggered in the backend for which the derivation hadn't been pushed to the cache yet, causing needless build failures. #314
-
A build error caused by a moved symbol in
cachix >= 0.7
. #363 -
A test that relied on
aeson
field order, which isn't stable. #352
-
The path to
secrets.json
is now configurable in the module or config file, using thesecretsJsonPath
setting, analogous toclusterJoinTokenPath
. -
aarch64-darwin
is now officially supported. -
All module settings options will be visible in the NixOS documentation. Some less-used settings were hidden, specifically the file path options that default to
staticSecretsDirectory + "/cluster-join-token.key"
, etc.
-
The flake packages and modules now link with Nix 2.4. Nix 2.3 support is still available via
packages.${system}.hercules-ci-agent-nix_2_3
, but will be removed in agent 0.9.x. -
No longer patch Boehm GC, staying closer to regular Nix and the Nixpkgs build of the
hercules-ci-agent
.
- Various fixes related to the upgrade to Nix 2.4.
hci
can now run in the effects sandbox
- Interrupt handling has been improved
- Preparations for the next Nix version
- #304,
message:epollControl: invalid argument (Bad file descriptor)
in effect task
- Attach user-defined labels to the agent, for retrieval via the API.
- Fix an issue where long-runnin nix methods weren't interrupted
- Fix error
mkdir /run/runc: permission denied
-
Hercules CI Effects, a new feature for running programs that interact with the real world, with useful features for continous deployment.
-
Effects only run after the build completes successfully
-
Effects are defined like a derivation, not unlike a Nix shell
-
Independent processes can run concurrently as distinct effects
-
No two commits in the same repo run effects at the same time; no need to worry about concurrency in deployment scripts
-
Effects each run in their own sandbox with access to network, Nix store, remote state file API and secrets
-
Secrets are configured locally on your agents, so you don't have to trust a third party with your cloud credentials
-
-
Hercules CI Agent is now a flake. The highlights are
nixosModules
overriding the NixOS-distributed module to the in-repo versionagent-profile
for agent machines, oragent-service
for just the service definition
packages
hercules-ci-cli
the user command line interfacehercules-ci-agent
for custom installation methods, etc
-
The
hci
command (flake:defaultApp
)hci login
to authenticate yourselfhci state
to work with Effects state fileshci effect
to run effects locally
-
Commit metadata as a
ci.nix
argument. Make yourci.nix
a function:{ src ? { ref = null; rev = null; }}: # rest of your ci.nix
src.ref
will have e.g.refs/heads/master
andrev
will have the git commit SHA. -
Shell derivations will only be built for their dependencies. Add a
mkShell
-based expression like you would add a derivation.This behavior can be requested explicitly for shells and non-shell derivations alike by appending
// { buildDependenciesOnly = true; }
to the attribute definition. -
Attributes can now be marked to require or ignore a build failure in the derivation it references directly. (see support#34)
-
concurrentTasks
now has a default,"auto"
for ease of setup and to help avoid underutilization.
-
The parent directory name will match the repo name support#40
-
Previously, lines from Nix's configured netrc file were ignored. Now they are appended to Hercules CI's netrc lines.
- Cachix caches without
signingKeys
will be pushed to, as part of the recently introduced write token feature (Cachix-managed signing keys)
0.7.5 - 2020-11-27
-
GHC 8.10.2 compatibility for NixOS unstable / NixOS 21.03
-
Build with cachix 0.5.1. Write token support is not included due to a break configuration semantics. It will be available in >= 0.8.
-
The in-repo expressions have upgraded their dependency to NixOS 20.09
-
Agent will now use
HERCULES_CI_API_BASE_URL
overHERCULES_API_BASE_URL
if set. -
Temporarily switch to cabal tooling due to breakage.
master
continues to be stack-based.
0.7.4 - 2020-08-25
-
Paths that are missing from the binary cache will be rebuilt. This affected agent 0.7 - 0.7.3 users with trusted user optimizations turned on, which is the default when using the NixOS or nix-darwin modules.
-
Prevent states where no progress can be made. One caused by a potential buildup of batched messages that may not fit within the timeout interval; the other a receive operation without a timeout during the initial socket handshake.
-
The log socket will remain open instead of reconnecting unnecessarily.
-
Add a safety measure to prevent unintended increases in workload in case Nix sees an opportunity for concurrency that was not intended by Hercules CI.
-
The NixOS module in the hercules-ci-agent repo now disables the upcoming module that is packaged upstream with NixOS.
The upstream module will configure fewer things for you, to be in line with normal NixOS expectations. Notably, it does not configure automatic garbage collection and it does not preconfigure NixOps keys deployment.
The configuration interface in the hercules-ci-agent repo will remain unchanged for
0.7
but0.8
will match the upstream interface.
0.7.3 - 2020-07-18
-
Evaluation log
-
Configurable log level via config file or
extraOptions
- Default log level is
InfoS
rather thanDebugS
0.7.2 - 2020-06-18
- cachix: 0.3.7 -> 0.3.8. Improves reliability through better retries and improves error reporting
0.7.1 - 2020-06-17
- Push to any nix store, including S3, using the
NixCache
kind inbinary-caches.json
-
Switch to Nixpkgs 20.03
-
Environment variables are now passed on to evaluation and build by default. This allows configuration to be passed to Nix without intervention from hercules-ci-agent.
-
trusted-user
is not a requirement and is configured automatically when using the NixOS module or nix-darwin module.
-
Agent will now reset its connection with hercules-ci.com when pings are not acknowledged in time.
-
Prevent running out of file descriptors by increasing the soft limit if possible.
- Agent process user must be in
trusted-users
. This is the case with the NixOS and nix-darwin module. Doing so is recommended for ease of use and performance but should not be a requirement.
-
Jobs can be cancelled
-
Build logs are streamed in realtime
-
The build log now has timestamps and color
-
Distributed builds performance has improved by fetching fewer derivations
- Extra nix options were not passed to the build process
0.6.6 - 2020-03-16
-
NixOS, nix-darwin modules: check the
nix-daemon
source and add option to patch an in-memory cache expiry issue causing errors in build clusters (of more than 1 machine). The module asks for confirmation. -
Manual action: if you are not using the provided module and you run the agent on more than one machine, and you use
nix-daemon
, to fix above issue, you need to:- either upgrade your system's Nix to a recent
master
or version 2.4.0 when released, - or apply this patch to your system's Nix installation: NixOS/nix#3405
- either upgrade your system's Nix to a recent
-
Cachix: 0.3.5 -> 0.3.7 to prevent uploading bad NARs in rare cases.
0.6.5 - 2020-03-07
- Work around a systemd behavior where it didn't restart the unit
--test-configuration
flag to validate the configuration without actually running the agent.
0.6.4 - 2020-03-06
- Fix a bug blocking evaluation when a store path is removed from cache or cache configuration is changed.
- Cached builds to speed up
aarch64-linux
agent deployments.
0.6.3 - 2020-02-19
-
Fix a concurrency problem causing not all evaluation events to be written to server when evaluation fails.
-
Fix evaluation errors triggered by build outputs go missing from cache, by requesting a forced rebuild.
-
Fix blocked shutdown on NixOS, fix agent status in dashboard, by stopping agent before network shutdown #195.
-
Fix upload of large outputs by using the correct http client manager for Cachix, removing a timeout.
- Agent will now try to verify that the nix-daemon has narinfo-cache-negative-ttl = 0. This is required for correct operation.
0.6.2 - 2020-01-30
- Update cachix to support the API change for the new CDN. This update is required for uploading sources and compressed outputs over 100MB in size. Please update.
0.6.1 - 2019-11-06
-
Fix token leak to system log when reporting an HTTP exception. This was introduced by a library upgrade. This was discovered after tagging 0.6.0 but before the release was announced and before moving of the
stable
branch. Only users of thehercules-ci-agent
master
branch and the unannounced tag were exposed to this leak. We recommend to follow thestable
branch. -
Temporarily revert a Nix GC configuration change that might cause problems until agent gc root behavior is improved.
0.6.0 - 2019-11-04
- Switch to Nix 2.3 and NixOS 19.09. You should update your deployment to reflect the NixOS upgrade, unless you're using terraform or nix-darwin, where it's automatic.
- Increased parallellism during push to cachix
- Switch to NixOS 19.09
- Enable min-free/max-free Nix GC
- Transient errors during source code fetching are now retried
- Fixed a bug related to narinfo caching in the context of IFD
- Fixed an exception when the root of ci.nix is a list, although lists are unsupported
0.5.0 - 2019-10-01
- Now deployable with terraform-hercules-ci
-
The
binary-caches.json
file can now be deployed like any other confidential file. Its contents are not required at module evaluation time any more. -
The
services.hercules-ci-agent.binaryCachesFile
option has been removed.NixOps users: rename to
deployment.keys."binary-caches.json".file
Others: remove your
binaryCachesFile
value. Make surebinary-caches.json
is deployed. -
The
binary-caches.json
file is now required. The empty object{}
is a valid file, but we highly recommend to configure a cache.
- The agent will now actually auto-restart when the secrets files change.
0.4.0 - 2019-08-30
- Support for import-from-derivation. See https://blog.hercules-ci.com/2019/08/30/native-support-for-import-for-derivation/ for details.
-
Report build failures and technical errors (misconfigurations, etc) separately
-
Remove HerculesScribe
-
Worker now uses structured logging (including worker pid, etc)
-
Disable parallel GHC GC to improve runtime performance
-
Bump Cachix to fix a few bugs (errors with too many derivations, performance fixes, etc.)
-
Modern BoehmGC initial settings for Nix memory limits
0.3.2 - 2019-08-11
- Deploying the agent from different system (darwin to linux) resulted into using the wrong executable
0.3.1 - 2019-08-07
-
Emit a log when evaluator starts to push to cachix
-
Increase attribute limit to 50k
-
Pin nixpkgs commit and speed up compilation via https://hercules-ci.cachix.org
-
Possible exception during evaluation was not propagated, resulting into lack of retries
-
#8: Refresh agent session on cluster join token change
-
Fix segfault on involved IFD project (remove a finalizer)
-
Cachix: fix a crash with a lot of attributes (when determining closure graph)
0.3.0 - 2019-07-05
-
Configuration of the agent is now done via
--config agent.toml
so all command line arguments were removed. -
NixOS-based deployments now require
enable
.services.hercules-ci-agent.enable = true;
-
All files are placed/expected in new locations that by default derive from the
baseDirectory
option in theagent.toml
file.You may remove
~/.hercules-ci-agent
and~/.local/share/hercules-ci-agent
after upgrading.
- Added retries to status reporting to fix potential inconsistencies on the service
-
Added Cachix support, for multi-agent and multi-platform support
-
Report derivation outputs with their size and hashes
-
Added Darwin support via nix-darwin
-
Support
requiredFeatures
attribute on derivations -
Hello and hearthbeat protocol, which will allow the service to be aware of how the agent is configured and when it's online.
0.2 - 2019-05-14
- use gitignore instead of nix-gitignore
- fix build on Darwin
- limit internal concurrency to max eight OS threads for beefier machines
- show version on
--help
- build against NixOS 19.03 as default
- propagate agent information to agent view: Nix version, substituters, platform and Nix features
0.1.1 - 2019-04-16
- Support ci.nix or nix/ci.nix along with default.nix
- Initial release