Release v1.5.0.1

This very minor release only affects the static trident executables produced for every release.

It introduces a distinction between pre-built X64 and ARM64 executables for macOS, where changes in the main processor architecture have recently rendered old builds invalid for new systems and vice versa.

That means the executable trident-macOS will henceforward not longer exist, but instead the executables trident-macOS-X64 and trident-macOS-ARM64.

In the past we have not explicitly documented changes in the compilation pipeline - v1.5.0.0, for example, came with a major overhaul of the pipeline - but in this case a small version bump seems to be in order to announce the split in available artefacts.

Release v1.5.0.0

This is a minor, but technically breaking release. It removes the example contributor Josiah Carberry from new packages created by trident init and trident forge

Previously every package created by init or forge included an example entry in the contributor field of the POSEIDON.yml file:

- name: Josiah Carberry
  email: carberry@brown.edu
  orcid: 0000-0002-1825-0097

This served the purpose of reminding users to actually set a contributor and giving an example how to do so. To simplify scripting with Poseidon packages we now remove this slightly gimmicky default.

To encourage setting the contributor field we instead introduce a reading/validation warning in case the contributor field is empty:

[Warning] Contributor missing in POSEIDON.yml file of package 2010_RasmussenNature-2.1.1

Release v1.4.1.0

This release adds an entirely new subcommand to merge two .janno files (jannocoalesce) and improves the error messages for broken .janno files.

Merging .janno files with jannocoalesce

The need for a tool to combine the information of two .janno files arose in the Poseidon ecosystem as we started to conceptualize the Poseidon Minotaur Archive. This archive will be populated by paper-wise Poseidon packages for which the genotype data was regenerated through the Minotaur workflow (work in progress). We plan to reprocess various packages that are already in the Poseidon Community Archive and for these packages we want to copy e.g. spatiotemporal information from the already available .janno files. jannocoalesce is the answer to this specific need, but can also be useful for various other applications.

It generally works by reading a source .janno file with -s|--sourceFile (or all .janno files in a -d|--baseDir) and a target .janno file with -t|--targetFile. It then merges these files by a key column, which can be selected with --sourceKey and --targetKey. The default for both of these key columns is the Poseidon_ID. In case the entries in the key columns slightly and systematically differ, e.g. because the Poseidon_IDs in either have a special suffix (for example _SG), then the --stripIdRegex option allows to strip these with a regular expression to thus match the keys.

jannocoalesce generally attempts to fill all empty cells in the target .janno file with information from the source. --includeColumns and --excludeColumns allow to select specific columns for which this should be done. In some cases it may be desirable to not just fill empty fields in the target, but overwrite the information already there with the -f|--force option. If the target file should be preserved, then the output can be directed to a new output .janno file with -o|--outFile.

Better error messages for broken .janno files

.janno file validation is a core feature of trident. With this release we try to improve the error messages for a two common situations:

1. Broken number fields. This can happen if some text or wrong character ends up in a number field.

So far the error messages for this case have been pretty technical. Here for example if an integer field is filled with 430;, where the integer number 430 is accidentally written with a trailing ;:

parse error (Failed reading: conversion error: expected Int, got "430;" (incomplete field parse, leftover: [59]))

The new error message is more clear:

parse error in one column (expected data type: Int, broken value: "430;", problematic characters: ";")

2. Inconsistent Date_*, Contamination_* and Relation_* columns. These sets of columns have to be cross-consistent, following a logic that is especially complex for the Date_* fields (see here).

So far any inconsistency was reported with this generic error message:

The Date_* columns are not consistent

Now we include far more precise messages, like e.g.:

Date_Type is not "C14", but either Date_C14_Uncal_BP or Date_C14_Uncal_BP_Err are not empty.

This should simplify tedious .janno file debugging in the future.

Release v1.4.0.3

This small release fixes a performance issue related to finding the latest version of all packages. The bug had severe detrimental effects on forge and fetch, which are now resolved.

We used this opportunity to switch to a new GHC version and new versions of a lot of dependencies for building trident.

Release v1.4.0.2

This release finally fully enables handling multiple Poseidon package versions with trident. It includes a significant overhaul of the selection language in forge and fetch with major changes in its implementation and, as a consequence, multiple (subtle, but strictly breaking) changes in its semantics.

Handling multiple package versions

The trident subcommands fetch, forge, genoconvert, list, rectify, survey and validate now by default consider all versions of each Poseidon package in the given base directories. Previously all of them only considered the latest versions. If this old behaviour is desired now, it can be enabled with the flag --onlyLatest for the subcommands genoconvert, list, rectify, survey and validate. fetch and forge now generally consider all package versions (if we ignore the selection language semantics) and summarize continues to consider only the latest ones.

Changes to the selection language

In the forge- and fetch-selection language it is now possible to specify which version of a Poseidon package should be forged/fetched by appending the version number after a minus: e.g. *2010_RasmussenNature-2.1.1*. This also works for the more verbose and precise syntax to describe individuals, e.g. <2010_RasmussenNature-2.1.1:Greenland_Saqqaq.SG:Inuk.SG>.

While implementing this change, we also reworked the entity selection logic. It now adheres to the following rules:

Inclusion queries

• *Pac1*: Select all individuals in the latest version of package "Pac1"
• *Pac1-1.0.1*: Select all individuals in package "Pac1" with version "1.0.1"
• Group1: Select all individuals associated with "Group1" in all latest versions of all packages
• <Ind1>: Select the individual named "Ind1", searching in all latest packages.
• <Pac1:Group1:Ind1>: Select the individual named "Ind1" associated with "Group1" in the latest version of package "Pac1"
• <Pac1-1.0.1:Group1:Ind1>: Select the individual named "Ind1" associated with "Group1" in the package "Pac1" with version "1.0.1"

Exclusion queries

• -*Pac1*: Remove all individuals in all versions of package "Pac1"
• -*Pac1-1.0.1*: Remove only individuals in package "Pac1" with version "1.0.1" (but leave other versions in)
• -Group1: Remove all individuals associated with "Group1" in all versions of all packages (not just the latest)
• -<Ind1>: Remove all individuals named "Ind1" in all versions of all packages (not just the latest).
• -<Pac1:Group1:Ind1>: Remove the individual named "Ind1" associated with "Group1", searching in all versions of package "Pac1"
• -<Pac1-1.0.1:Group1:Ind1>: Remove the individual named "Ind1" associated with "Group1", but only if they are in "Pac1" with version "1.0.1"

Missing (or mis-spelled) entities in a selection-set lead to errors now.

If the forge entity list starts with a negative entity, or if the entity list is empty, forge will still implicitly assume you want to include all individuals from only the latest packages found in the baseDirs.
Likewise, trident fetch --downloadAll considers only latest versions now.

The specific individual selection syntax (with -<Pac1-1.0.1:Group1:Ind1>) does not perform automatic duplicate resolution any more. If there is another <Ind1> somewhere within the selected entities, then this will cause an error that has to be resolved manually by adjusting the selection.

Minor additional changes

• The Web API and the list subcommand now return an extra, boolean field/column isLatest to point out if an entity (individual, group, package) is from the latest package version.
• list now also returns column headers with the --raw flag. If they are not desired, then they have to be filtered out manually on the command line (e.g. with trident list ... | tail -n+2).
• Duplicate individuals in a package collection do not anymore lead to errors. Instead, only a selection for forge (and externally also in xerxes), if resulting in multiple individuals in the selection, will lead to errors. validate will also fail by default, except --ignoreDuplicates is set.

Release v1.3.0.4

This is a significant release with a breaking change, multiple new features and a number of minor fixes and improvements.

rectify replaces update

The subcommand update was renamed to rectify to better express its purpose. The name update suggested that this command could effectively migrate packages from one Poseidon version to the next, which was never the case. Structural and semantical changes to the package always have to be performed by the user through other means, so usually by manually editing the respective package files. rectify only helps to adjust a number of parameters (mostly in the POSEIDON.yml file) after the changes have been applied: It updates checksums, iterates version numbers, adds contributors and appends logging information to CHANGELOG files. Despite this limitation it is still a valuable tool, especially for the management of large package archives, where structural changes are often applied to many packages at once, all requiring "rectification" in the end.

rectify doesn't only introduce a different name, it also features a different interface. While update was a catch-all procedure with an opinionated, default behaviour that could partially be adjusted with various flags, rectify follows a much more transparent opt-in philosophy. The new interface allows to precisely choose which aspects of a package should be updated.

⚠️ Please note that certain changes like incrementing the version number or adding a logText therefore do not happen automatically any more in rectify. They have to be requested explicitly!

Here is the new command line documentation of rectify:

Usage: trident rectify (-d|--baseDir DIR) [--ignorePoseidonVersion]
                       [--poseidonVersion ?.?.?]
                       [--packageVersion VPART [--logText STRING]]
                       [--checksumAll | [--checksumGeno] [--checksumJanno]
                         [--checksumSSF] [--checksumBib]]
                       [--newContributors DSL]

  Adjust POSEIDON.yml files automatically to package changes

Available options:
  -h,--help                Show this help text
  -d,--baseDir DIR         A base directory to search for Poseidon packages.
  --ignorePoseidonVersion  Read packages even if their poseidonVersion is not
                           compatible with trident.
  --poseidonVersion ?.?.?  Poseidon version the packages should be updated to:
                           e.g. "2.5.3".
  --packageVersion VPART   Part of the package version number in the
                           POSEIDON.yml file that should be updated: Major,
                           Minor or Patch (see https://semver.org).
  --logText STRING         Log text for this version in the CHANGELOG file.
  --checksumAll            Update all checksums.
  --checksumGeno           Update genotype data checksums.
  --checksumJanno          Update .janno file checksum.
  --checksumSSF            Update .ssf file checksum
  --checksumBib            Update .bib file checksum.
  --newContributors DSL    Contributors to add to the POSEIDON.yml file in the
                           form "[Firstname Lastname](Email address);...".

serve now provides different package archives and list and fetch can access them

trident serve, so the subcommand behind the server providing the Poseidon Web API, can now host packages from multiple named archives in parallel. This works through a modified -d interface on the command line and a new option ?archive=... in the Web API. The client commands fetch and list can request information and package download from these different archives with a new option --archive. If --archive (or ?archive=... in the http request) are not given, then the server falls back to a default archive (the first in -d).

See the Poseidon public archive and Web API documentation for the concrete consequences of this new feature.

validate can now check individual files

The validate subcommand is no longer confined to validating entire poseidon packages. It can still very much do so with -d, where -- just as before -- a number of optional flags can be used to control the exact behaviour. This release, in fact, adds the new options --ignorePoseidonVersion and --ignoreChecksums here. But validate can now also read, parse and thus check individual files: POSEIDON.yml files, genotype data files, .janno files, .ssf files or .bib files. This is tremendously useful for building packages step-by-step, e.g. in automatic pipelines.

⚠️ Please note that these individual file checks naturally do not include the cross-file checks (e.g. publication keys across .janno and .bib). These are only available in the full package validation process.

Here is the new command line documentation of validate:

Usage: trident validate ((-d|--baseDir DIR) [--ignoreGeno] [--fullGeno]
                          [--ignoreDuplicates] [-c|--ignoreChecksums]
                          [--ignorePoseidonVersion] |
                          --pyml FILE | (-p|--genoOne FILE) | --inFormat FORMAT
                          --genoFile FILE --snpFile FILE --indFile FILE |
                          --janno FILE | --ssf FILE | --bib FILE) [--noExitCode]

  Check Poseidon packages or package components for structural correctness

Available options:
  -h,--help                Show this help text
  -d,--baseDir DIR         A base directory to search for Poseidon packages.
  --ignoreGeno             Ignore snp and geno file.
  --fullGeno               Test parsing of all SNPs (by default only the first
                           100 SNPs are probed).
  --ignoreDuplicates       Do not stop on duplicated individual names in the
                           package collection.
  -c,--ignoreChecksums     Whether to ignore checksums. Useful for speedup in
                           debugging.
  --ignorePoseidonVersion  Read packages even if their poseidonVersion is not
                           compatible with trident.
  --pyml FILE              Path to a POSEIDON.yml file.
  -p,--genoOne FILE        One of the input genotype data files. Expects .bed,
                           .bim or .fam for PLINK and .geno, .snp or .ind for
                           EIGENSTRAT. The other files must be in the same
                           directory and must have the same base name.
  --inFormat FORMAT        The format of the input genotype data: EIGENSTRAT or
                           PLINK. Only necessary for data input with --genoFile
                           + --snpFile + --indFile.
  --genoFile FILE          Path to the input geno file.
  --snpFile FILE           Path to the input snp file.
  --indFile FILE           Path to the input ind file.
  --janno FILE             Path to a .janno file.
  --ssf FILE               Path to a .ssf file.
  --bib FILE               Path to a .bib file.
  --noExitCode             Do not produce an explicit exit code.

Other, minor changes

• Fixed the behaviour of forge when combining .bib files. Publication duplicates are now properly removed upon merging and the output is alphabetically sorted.
• Added a global option --debug, which is short for --logMode VerboseLog.
• Refactored summarise to make the resulting counts more accurate. Some variables in the output table have been renamed as well.
• Fixed the behaviour of chronicle when updating a chronicle file (with -u): The lastModified field is now only touched if there is actually a change in the package list.
• Some cleaning of the general trident command line documentation: Added meaningful meta variables to all arguments.
• Shortened the default command line output of fetch to make it more readable.
• Slightly better error handling for failed http requests in fetch and list --remote.

Release v1.2.1.0

This release does not include changes for trident end users.

It adds two new subcommands for (public) archive management, but they are only relevant from a developer's perspective: chronicle creates or updates a dedicated .yml file to document version iterations of Poseidon packages in a Git-managed archive, and timetravel recovers package iterations based on this file to (re)construct said archive from the source repository.

Just as serve, both new subcommands will be omitted in the command line help.

Release v1.2.0.0

This release comes with a massive rewrite of the server-client infrastructure, so the code behind the API to download and list packages from our public data archives.

The server is now implemented as a (hidden) subcommand of trident: serve. It returns helpful error messages, if an incompatible version of trident tries to connect to it. And it is now capable of serving multiple (so not just the latest, but also older) versions of one package, which is an important step towards computational reproducibility of Poseidon-based pipelines.

All Server-APIs except for zip_file now return a complex JSON datatype with server messages and a payload. The messages contain standard messages like a greeting and in the future perhaps also deprecation warnings. Some APIs also provide information or warnings in the server messages.

All APIs except for zip_file also accept an additional parameter ?client_version=X.X.X, so that the server may in the future respond to old clients that an update is needed in order to understand the API. Our trident list --remote functionality already makes use of this.

Here are the individual APIs:

• https://server.poseidon-adna.org/packages: returns a list of all packages.
• https://server.poseidon-adna.org/groups: returns a list of all groups.
• https://server.poseidon-adna.org/individuals: returns a list of all individuals.
• https://server.poseidon-adna.org/zip_file/<package_name>?package_version=1.0.1: returns a zip file of the package with the given name and the given version. If no version is given, it returns the latest.

The client subcommands fetch and list can not yet make full use of this new API in this release, because they lack an interface to request specific package versions. This will be added in a future release. But the output of both subcommands already differs from the previous implementation:

• fetch now appends the package version to the directory name when downloading a package. Previously trident fetch -d . -f "*2010_RasmussenNature*" yielded a directory named 2010_RasmussenNature with the package data, but now it creates 2010_RasmussenNature-2.0.1 (or whatever is the latest version of this package in the archive).
• fetch does no longer have an option --upgrade, since the new behaviour respects different versions to live side by side in different directories. If users whish to remove old versions, they should do so manually.
• list lists not just data (package, groups, individuals) for the latest version of a package, but for all versions. The package version became an explicit output column.

As before, forge and the other subcommands keep ignoring multiple package versions for now, and only read the latest.

The new server is available at a new URL (https://server.poseidon-adna.org), but the old version at (https://c107-224.cloud.gwdg.de) will also keep running for now. New releases of trident (v1.2.0.0+) will by default use the new server, while older versions must connect to the old one.

Release v1.1.12.0

This release implements the changes necessary for the Poseidon schema v2.7.1. That mostly means that the constraints on several .ssf file columns previously considered mandatory and unique were lifted.

Beyond that a number of type constraints specified already in Poseidon v2.7.0 for the .ssf file were finally implemented in poseidon-hs. A broken file will, thus actually be flagged upon reading if it violates the following requirements:

• .ssf columns that include Accession_IDs have to feature the correct and valid Accession_IDs according to INSDC specification.
• .ssf columns with dates have to be valid dates of the form YYYY-MM-DD.
• .ssf columns featuring MD5 hashes require entries with exactly 32 hex-digits.

Both for the .janno and the .ssf file we elevated the log level of the common broken lines error from debug to error. This makes these errors more prominent and more easy to resolve.

Release v1.1.11.4

This release fixes a core issue in the implementation of Poseidon v2.7.0, where multiple columns of the .ssf file where not defined correctly as list columns. Poseidon v2.7.0 is in itself deprecated, though, and will be replaced as soon as possible with an updated version. This trident release exists thus mainly to provide a working implementation of 2.7.0 for future reference.

Beyond this change in functionality, this release also includes heavy refactoring in the survey subcommand, the golden test infrastructure and the overall version of Haskell poseidon-hs and trident are built with. These changes should not have any user-facing consequences.