Impact
This security audit describes a vulnerability in the in-toto architecture, which allows an attacker, who controls the product in transit, to compromise the whole supply chain and stay undetected.
This is achieved by modifying the product in transit in such a way that the product itself can compromise a subsequent functionary, in order to (1) introduce malicious artifacts into the supply chain and (2) fake the corresponding link metadata.
To stay undetected the material hashes in the link metadata must be faked to match the original product, as reported by the preceding link metadata, and not the tampered one, which was actually used as material.
While in-toto is not designed to protect against the compromise of individual functionaries, beyond supporting link metadata thresholds, this specific case of functionary compromise needs to be taken seriously, because the in-toto threat model indeed claims that it protects against unauthorized modification of the product in transit.
An example of this compromise uses the following simple supply chain scenario:
- A 'tag' step has source code files as products.
- A 'build' step has source code files as materials and a binary as a product.
- A layout defines that materials of 'build' must match products of 'tag', and that the final product must match the product of 'build'.
This attack is successful when:
- The attacker can modify the source code between ‘tag’ and ‘build’ steps. Some examples of how this may occur are by compromising the functionary performing the tag or by being a man-in-the-middle on the network if insecure protocols are used.
- The functionary, who builds the attacker-modified source code, unwittingly produces
- a compromised product, and
- incorrect link metadata, which lists the benign source code files as materials, instead of the tampered ones that were actually used.
- No other step in the supply chain detects the mismatch between the products reported by the 'tag' step and the tampered ones.
Note that a malicious functionary could always have performed whatever malicious action they like on the products they produce. However, this vulnerability gives them the additional ability to compromise later functionaries in some cases. This can not only hide the location of the compromise, but may also be used to cause later functionaries to fail to perform their steps (e.g., validate, test, fuzz, package, etc.) correctly.
Patches
Workarounds
Partial verification of layouts
Security auditors recommend to "have each functionary verify the steps before to ensure all actions are performed on trusted data".
The idea of partial verification of layouts is not new to the in-toto team (see references), but has intentionally not been adopted, in order to not sacrifice generality or require drastic specification changes, because:
- The order of steps in an in-toto layout does not necessarily reflect the order in which steps are carried out.
- The artifact rules of any step may reference the artifacts of any other step in the supply chain, not only adjacent ones.
- Functionaries otherwise don't need to have access to any other link or layout metadata.
- The only way of matching local artifacts with the artifacts reported from link metadata is by the use of inspections. However, the layout cannot define inspections to be executed mid-way.
The volume of required changes and workaround in order to solve the reported issue with partial layout verification makes this solution undesirable.
Custom verification of preceding links
Alternatively, a non-in-toto-compliant functionary-side verification could be developed, which considers only the preceding link metadata file, and matches its products to the local artifacts used by the functionary. This feature was added strictly for convenience to the in-toto reference implementation.
However, relying on this feature is not recommended as it requires an out-of-band negotiation about what the preceding link metadata is, who is authorized to sign it, and how it relates to the materials the verifying functionary is operating on.
These prerequisites amount to a similar set of policies as an in-toto layout would cover. Creating a custom protocol to promote and verify these policies seems undesirable.
Verification of partial layouts
A non-custom and specification compliant solution would be that project owners provide additional partial layouts to each functionary who can then perform a full in-toto-compliant verification prior to performing their functionary task.
This is already possible without specification or implementation change, and may be listed as a solution to the described issue in in-toto usage documentation. However, acknowledging the complexity of creating a single layout for a project, UX implications of this recommendations should be considered carefully.
Encourage sandboxing of functionaries (recommended)
There is an orthogonal solution, which does not require per-functionary verification of preceding subgraphs of the supply chain. Instead this solution mitigates the impact of a functionary compromise. That is, functionaries should be encouraged to strictly separate the routines that generate and sign link metadata, and the ones that run functionary specific code.
This recommendation also aligns with the SLSA Level 3 requirement for Provenance generation that "Provenance is Non-forgeable" and can be applied without changes to the in-toto specification.
We suggest adopting this solution by acknowledging the described attack vector and recommending the usage of sandboxing technology in in-toto documentation. In addition, we suggest documenting usage recommendations and security pitfalls for in-toto tools that generate and sign link metadata.
For instance, the in-toto-run CLI/API generates and signs link metadata in the parent process, and executes untrusted functionary code in a subprocess. This means that the untrusted code can inherit the privileges of the trusted code, e.g. to access files or environment variables, unless the subprocess is additionally isolated, e.g. using container technology.
Similarly, the in-toto-record CLI/API, uses separate processes to generate and sign link metadata, and to run untrusted functionary code. This is susceptible to the same pitfalls as in-toto-run, if the processes are run with the same privileges.
In both cases, we recommend that one drops all possible privileges before executing functionary specific code. This is especially critical for any functionary code that executes portions of its materials.
References
"Enable partial verification" in-toto/in-toto#116
Impact
This security audit describes a vulnerability in the in-toto architecture, which allows an attacker, who controls the product in transit, to compromise the whole supply chain and stay undetected.
This is achieved by modifying the product in transit in such a way that the product itself can compromise a subsequent functionary, in order to (1) introduce malicious artifacts into the supply chain and (2) fake the corresponding link metadata.
To stay undetected the material hashes in the link metadata must be faked to match the original product, as reported by the preceding link metadata, and not the tampered one, which was actually used as material.
While in-toto is not designed to protect against the compromise of individual functionaries, beyond supporting link metadata thresholds, this specific case of functionary compromise needs to be taken seriously, because the in-toto threat model indeed claims that it protects against unauthorized modification of the product in transit.
An example of this compromise uses the following simple supply chain scenario:
This attack is successful when:
Note that a malicious functionary could always have performed whatever malicious action they like on the products they produce. However, this vulnerability gives them the additional ability to compromise later functionaries in some cases. This can not only hide the location of the compromise, but may also be used to cause later functionaries to fail to perform their steps (e.g., validate, test, fuzz, package, etc.) correctly.
Patches
Workarounds
Partial verification of layouts
Security auditors recommend to "have each functionary verify the steps before to ensure all actions are performed on trusted data".
The idea of partial verification of layouts is not new to the in-toto team (see references), but has intentionally not been adopted, in order to not sacrifice generality or require drastic specification changes, because:
The volume of required changes and workaround in order to solve the reported issue with partial layout verification makes this solution undesirable.
Custom verification of preceding links
Alternatively, a non-in-toto-compliant functionary-side verification could be developed, which considers only the preceding link metadata file, and matches its products to the local artifacts used by the functionary. This feature was added strictly for convenience to the in-toto reference implementation.
However, relying on this feature is not recommended as it requires an out-of-band negotiation about what the preceding link metadata is, who is authorized to sign it, and how it relates to the materials the verifying functionary is operating on.
These prerequisites amount to a similar set of policies as an in-toto layout would cover. Creating a custom protocol to promote and verify these policies seems undesirable.
Verification of partial layouts
A non-custom and specification compliant solution would be that project owners provide additional partial layouts to each functionary who can then perform a full in-toto-compliant verification prior to performing their functionary task.
This is already possible without specification or implementation change, and may be listed as a solution to the described issue in in-toto usage documentation. However, acknowledging the complexity of creating a single layout for a project, UX implications of this recommendations should be considered carefully.
Encourage sandboxing of functionaries (recommended)
There is an orthogonal solution, which does not require per-functionary verification of preceding subgraphs of the supply chain. Instead this solution mitigates the impact of a functionary compromise. That is, functionaries should be encouraged to strictly separate the routines that generate and sign link metadata, and the ones that run functionary specific code.
This recommendation also aligns with the SLSA Level 3 requirement for Provenance generation that "Provenance is Non-forgeable" and can be applied without changes to the in-toto specification.
We suggest adopting this solution by acknowledging the described attack vector and recommending the usage of sandboxing technology in in-toto documentation. In addition, we suggest documenting usage recommendations and security pitfalls for in-toto tools that generate and sign link metadata.
For instance, the in-toto-run CLI/API generates and signs link metadata in the parent process, and executes untrusted functionary code in a subprocess. This means that the untrusted code can inherit the privileges of the trusted code, e.g. to access files or environment variables, unless the subprocess is additionally isolated, e.g. using container technology.
Similarly, the in-toto-record CLI/API, uses separate processes to generate and sign link metadata, and to run untrusted functionary code. This is susceptible to the same pitfalls as in-toto-run, if the processes are run with the same privileges.
In both cases, we recommend that one drops all possible privileges before executing functionary specific code. This is especially critical for any functionary code that executes portions of its materials.
References
"Enable partial verification" in-toto/in-toto#116