- A single test log is generated by a job which may consist of more than one step (only one of which creates the log)
- Several jobs (each generating its own test log) may be running in a run.
- A CI service starts a run when there is new code to be tested.
- Sometimes a CI service configures different jobs in different places, but this doesn't affect the classification
- "time" refers to a specific date & time; "duration" refers to an amount of time (a number of seconds). Field names ending in "time" or "duration" are considered to contain such information about the test run.
Using these data types, where appropriate, will make analysis of data easier.
- All metadata is stored as strings, but some only ever contain digits so they can be treated as integers.
- Times are stored as POSIX-style integer seconds past the epoch (Jan. 1, 1970)
- Durations are stored in integer microseconds
- Booleans are stored as strings with "yes" or "no"
- Arrays are stored as strings with array values separated with a special character, usually a space
- Don't include any information that you don't want made public. This is usually not a problem because the data will probably all be derived from public log files.
- Try to keep data mostly unchanged from its original form in the name of data preservation. It can always be massaged later. Exceptions are made for times and durations which should always be stored in the standard forms. In some cases, it may make sense to store an altered version in a separate field for ease of use. In other cases, altering the format of a field may make sense if it is information-preserving and if the alternate is easier to work with (like times and durations).
- Because of the previous rule, some fields may have different values but the same meaning. The interpretation of some fields may need to be gated by the contents of another field. e.g. the format of the "compiler" field might depend on the "buildsystem" field, or the values in the "ciarch" field might depend on the "origin" field.
- Boolean fields should be defined so that the complete absence of the field in the metadata should be treated as "no".
- Each test name for a checkrepo must refer to a single test, e.g. there may not be two completely different tests named "1". This does not mean that the same test may not be run under different conditions that test different things, in which case there should be a way to differentiate the two conditions in the metadata.
These are the mandatory metadata fields.
An indication of which service the test log originated from. Here are the supported values:
- appveyor: Appveyor
- azure: Azure Pipelines
- circle: Circle CI
- cirrus: Cirrus
- curlauto: Curl Autobuild system
- gha: GitHub Actions
- local: a local build
Like sourcerepo, but for the repository for which the test run was made. In
the case of a GitHub pull request, this is the repo to which the pull request
is being made. This must be the canonical URL for the repo, especially with
regard to a trailing slash.
Identifier of the account/project at the CI host. This must be omitted if it is not relevant to the CI service in use.
A unique identifier for this test run. This only needs to be unique relative to
origin, account (if found) and checkrepo. This field is used to
de-duplicate test runs so the same log file isn't accidentally imported
multiple times. TODO: the problem of re-runs is currently ignored
This is a unique identifier for the CI job that created this test log (and no other similar test logs). It can be used to compare subsequent test runs (on later commits). These are important points to consider in choosing a value:
- it only needs to be unique within the same 'origin',
accountandcheckrepo - there may be at most one type of test log (type parsed by one parser) associated with the identifier
- it should not be able for a user to create accidental collisions (e.g. by creating a new, carefully-crafted job name)
- it isn't really intended for human use, but only for machine correlation
- this is not to say the job is necessarily idempotent and unchangeable, but it's intended to stay the same run after run so users can see changes to the results over time
When the test run was requested to start running (in seconds since 1970). If this is not available, then runstarttime can be supplied instead. If this is not available, then runfinishtime can be supplied instead. One of those three is mandatory and is used as a broad selector for test runs (e.g. "all runs in the last 7 days").
These are some recommended metadata fields.
When the test run started running (in seconds since 1970).
When the test run finished running (in seconds since 1970).
When the job started running (in seconds since 1970).
When the job finished running (in seconds since 1970).
An identifier of where the source being tested came from. This might be e.g. a URL to a GitHub repo. In the case of a GitHub pull request, this is the forked repo. This must be the canonical URL for the repo, especially with regard to a trailing slash.
Identifier of the source being tested. This is relative to sourcerepo. This
field may end up becoming mandatory, since knowing the source is essential to
understanding the test results.
Identifier of the branch of the source being tested. This is relative to sourcerepo.
If jobs are configured in multiple places, this is the name of the place where this job is configured.
If jobs are configured in multiple places, this is the location of the place where this job is configured, like a file path.
The job name.
If this job contains multiple steps, this is the step name that generated the log.
A URL to a page giving more details of the results of this test run.
The kind of test log that was parsed to obtain these results.