Tap 7: Conformance testing #19
Conversation
Edit list of authors to remove Justin (who is only reviewing and giving feedback to the TAP).
|
@awwad @JustinCappos Can you please review TAP 7 when you get the chance? I will focus on adding some of the more important attacks and unit tests that remain, and verify that |
tap7.md
Outdated
| [Conformance testing](https://en.wikipedia.org/wiki/Conformance_testing) can | ||
| determine whether an implementation meets the requirements set by a | ||
| specification. A tool that helps developers and users test that an | ||
| implementation bahaves according to the TUF specification does not presently |
tap7.md
Outdated
| # Motivation | ||
|
|
||
| An adopter of the framework, say a developer that has written an implementation | ||
| in language X, can currently test for conformance by (1) verifying that |
There was a problem hiding this comment.
"can currently test for conformance by [doing 1 and/or 2]" is misleading. 1 would not test for conformance. I suggest instead:
"Adopters of the framework who have written implementations have tried testing for conformance by (1) ... "
tap7.md
Outdated
| behavior when the generated metadata is updated. In the second case, the | ||
| implementation is said to be conformant depending on how thoroughly the unit | ||
| tests are reproduced in X. There are bound to be inconsistencies between both | ||
| sets of unit tests, so a single tool is preferable. A single tool for |
There was a problem hiding this comment.
consider:
"There are bound to be inconsistencies between both sets of unit tests, and improvements in TUF testing or changes to TUF would result in a need for implemeters to add test code in parallel, so a single tool is preferable."
tap7.md
Outdated
| # Rationale | ||
|
|
||
| Developers of an implementation who wish to undergo conformance testing are | ||
| required to provide a program, or script, that accepts a specific set of |
There was a problem hiding this comment.
This implies that we will be doing the conformance testing for them, which is not how we expect to do this (though we're happy to help when possible). Instead, consider:
"Developers of an implementation can conformance test a program or script intended to perform secure updates that accepts a specific set of command-line arguments."
tap7.md
Outdated
| Developers of an implementation who wish to undergo conformance testing are | ||
| required to provide a program, or script, that accepts a specific set of | ||
| command-line arguments. The program should be able to perform a secure update. | ||
| A fixed set of arguments are needed so that conformance testing is consistent |
tap7.md
Outdated
| A fixed set of arguments are needed so that conformance testing is consistent | ||
| across different programs. The conformance tester also requires a minimum | ||
| number of arguments so that it can thoroughly cover all potential outcomes that | ||
| it wishes to test. It should be noted, however, that this program does not |
There was a problem hiding this comment.
Instead of the sentence that begins with "It should be noted...", consider:
"The implementers' updater may operate in a different mode than it would in production so as to generate interpretable error codes and accept command line arguments that the conformance tester would provide."
tap7.md
Outdated
| necessarily have to be the updater used in production, only that it should | ||
| function as defined in this TAP for conformance testing. | ||
|
|
||
| The program itself accepts a command-line argument that indicates the target |
There was a problem hiding this comment.
"An implementation to be tested will need to accept ..."
tap7.md
Outdated
| file to download when the program initiates an update, the location of a TUF | ||
| repository that satisfies requests for metadata and targets, and where | ||
| downloaded metadata files and the target file are saved when the program | ||
| initiates an update. Specifically, the program can be called as follows: |
There was a problem hiding this comment.
Specifically, the implementation needs to accept arguments as follows:
Show example Python implementation is wrapped in a script to exit with the expected return codes and how the command-line options are parsed.
Conflicts: tap7.md
tap7.md
Outdated
| (6) malicious mirrors | ||
| (7) mix-and-match | ||
| (8) rollback | ||
| (9) key compromise. |
There was a problem hiding this comment.
What is a "key compromise" attack and what does it mean to block it?
There was a problem hiding this comment.
An attacker tries to use a compromised key that has since been revoked. Do you have a suggestion for saying that in a concise way?
"Recover from key compromise?"
There was a problem hiding this comment.
It's not an attack, so maybe I should expand the description of the list.
There was a problem hiding this comment.
The SECURITY.md page uses "Vulnerability to key compromises." I should use that and say "attacks and weaknesses" in the description of the list.
https://github.com/theupdateframework/tuf/blob/develop/SECURITY.md
tap7.md
Outdated
| attacks on the updater are present, as defined later in the | ||
| *Specification* section of this TAP. Note that the conformance tester provides | ||
| the remote files (i.e., repository files) specified on the command-line, and so | ||
| it can test for various conditions and input metadata. |
There was a problem hiding this comment.
The second clause doesn't seem to follow, or at least I don't follow the connection. Can you restate this sentence?
|
I was going to take another look at this later in the afternoon. It looks
like you guys are in the middle of making some adjustments. Should I wait a
bit?
Lois
…On Tue, May 9, 2017 at 2:01 PM, Sebastien Awwad ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In tap7.md
<#19 (comment)>
:
> +
+In turn, the conformance tester tool executes this command when it runs its suite of
+tests, which will assess things like validation of the
+metadata downloaded by the updater, and verification that the following attacks are
+blocked:
+
+```
+(1) Arbitrary installation
+(2) endless data
+(3) extraneous dependencies
+(4) fast-forward
+(5) indefinite freeze
+(6) malicious mirrors
+(7) mix-and-match
+(8) rollback
+(9) key compromise.
What is a "key compromise" attack and what does it mean to block it?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#19 (review)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AN5qX59dNPoxDjdQN69F0cPcGlEZdnLOks5r4KoOgaJpZM4M6PYc>
.
|
tap7.md
Outdated
| populates the directory containing the remote repository files, and executes | ||
| the update command. The updater should load *tmp/metadata/root.json* (or the | ||
| appropriate path), refresh metadata accordingly, and lastly fetch the | ||
| requested update file . |
There was a problem hiding this comment.
I think this is an unclear description. It might help to refer to verified and unverified metadata directories (or perhaps client and repository)? The root file goes into the client's verified metadata directory, is a file the client is taken to have started off with. Repository files sit in another directory.
tap7.md
Outdated
| 4 | ||
| ``` | ||
|
|
||
| As before, the conformance tool is able to use this excution of the script to |
tap7.md
Outdated
| framework where metadata or update files are not saved to a file system on the | ||
| device. In this case, the developer or user running the conformance tests | ||
| would need to arrange for the files that are requested and stored by the device | ||
| to be saved to directories that *can* be accessed by the conformance tests. |
There was a problem hiding this comment.
I suggest adding "unmodified' -- "to be saved unmodified". The checks in the plan involve hashing the files to establish equality with the metadata files provided by the repository. (I still don't like this, but if we're doing it rather than only relying on return codes and the existence of target files, we should make it clear what the constraints are. For example, in the TUF reference implementation, you couldn't just write roledb contents to file, as those have had signatures stripped from them. The data written to disk has to be identical to the data drawn from the repository and validated in order for the hash check to work as a means of determining whether or not the correct metadata has been obtained and validated.)
"Unmodified" or similar additions may need to be added elsewhere, too; I'm not sure.
tap7.md
Outdated
| file(s) that the updater eventually trusts, and not that these files are | ||
| "installed" in some particular manner. Additionally, the conformance tests must | ||
| confirm the updater's ability to detect the attacks covered in the | ||
| specification (with the exlusion of the slow retrieval attack). |
tap7.md
Outdated
| and then manipulating the rate of transfer. Regardless of the transport | ||
| mechanism used, developers should take care to prevent their updaters from | ||
| being vulnerable to such attacks, which can happen before any data is | ||
| transferred, or after the it has begun. |
There was a problem hiding this comment.
typo - "after the it has begun"
|
Up to you, Lois - the comments I left aren't that extensive. |
|
Cool. I'm finishing up something else right now, but will get to this in an
hour or so.
Lois
…On Tue, May 9, 2017 at 2:19 PM, Sebastien Awwad ***@***.***> wrote:
Up to you, Lois - the comments I left aren't that extensive.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#19 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AN5qXz8fdaRorVNx37ZMyrRz39K-7Dt_ks5r4K4zgaJpZM4M6PYc>
.
|
Fix description of attacks protected against
Fix the second clause of the paragraph that explains how metadata is generated by the conformance tool
I made my first step in terms of re-organizing the document. I moved up the "Specification Heading" and then added some tentative subheads to this section. I'm not 100 percent sure I made these subhead breaks correctly, and I believe there might be some repetition that could be removed. I am by no means finished...I haven't even begun to think about the example section. But if you see things that you feel are flat out wrong, by all means correct them. Tomorrow, I'll take a first crack at the example section.
|
@jhdalek55 |
|
Not to my knowledge. I've been working on a cloned copy off Git Desktop. I
committed my last set of corrections, but I don;t recall doing anything
beyond that.
Lois
…On Mon, May 15, 2017 at 12:02 PM, Vladimir Diaz ***@***.***> wrote:
@jhdalek55 <https://github.com/jhdalek55>
Did you accidentally merge this pull request?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#19 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AN5qXwfYSKV7gFOvpfid6rshKlHpBvMRks5r6HcIgaJpZM4M6PYc>
.
|
|
It's not a problem, but it appears that you accidentally merged the TAP 7 pull request into the "master" branch here, and subsequent commits related to TAP 7 were then committed to the "master" branch: I've changed it so that PRs cannot be accidentally merged until they've been fully reviewed. You can continue making edits to TAP 7 on the "master" branch. We'll just update the status of the TAP 7 file once it's accepted. |
|
Ok. I'm sorry about that, Vlad.
Lois
…On Mon, May 15, 2017 at 3:12 PM, Vladimir Diaz ***@***.***> wrote:
It's not a problem, but it appears that you accidentally merged the TAP 7
pull request into the "master" branch here
<6abdb7a>,
and subsequent commits related to TAP 7 were then committed to the "master"
branch:
https://github.com/theupdateframework/taps/commits/master
I've changed it so that PRs cannot be accidentally merged until they've
been fully reviewed.
You can continue making edits to TAP 7 on the "master" branch. We'll just
update the status of the TAP 7 file once it's accepted.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#19 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AN5qX-srcObDlAzL-p-VzLi-ufFyesGoks5r6KOkgaJpZM4M6PYc>
.
|
|
I proofread TAP 7 again. I followed your instructions and continued working
on the "master." I committed and it opened a separate branch.
I think this is now in pretty good shape, except that it needs the example
scenarios and code. So, unless you make very large changes, you probably
won't need me on this anymore (so I can't mess up your git files).
Do let me know if and when you want me to look at TAP8 or 9.
Lois
…On Mon, May 15, 2017 at 3:17 PM, Lois A DeLong ***@***.***> wrote:
Ok. I'm sorry about that, Vlad.
Lois
On Mon, May 15, 2017 at 3:12 PM, Vladimir Diaz ***@***.***>
wrote:
> It's not a problem, but it appears that you accidentally merged the TAP 7
> pull request into the "master" branch here
> <6abdb7a>,
> and subsequent commits related to TAP 7 were then committed to the "master"
> branch:
> https://github.com/theupdateframework/taps/commits/master
>
> I've changed it so that PRs cannot be accidentally merged until they've
> been fully reviewed.
>
> You can continue making edits to TAP 7 on the "master" branch. We'll just
> update the status of the TAP 7 file once it's accepted.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <#19 (comment)>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AN5qX-srcObDlAzL-p-VzLi-ufFyesGoks5r6KOkgaJpZM4M6PYc>
> .
>
|
|
Justin,
What do you think of the revised TAP 7?
I think this is now in pretty good shape, except that it needs the example
scenarios and code.
I did add example scenarios and code. For instance, the modifying an
implementation to exit with expected return codes
<https://github.com/theupdateframework/taps/blob/fd62820169718886b1071ef5668bd2bcc76b32ff/tap7.md#modifying-an-implementation-to-exit-with-expected-return-codes>
subsection is covered here:
https://github.com/theupdateframework/taps/blob/fd62820169718886b1071ef5668bd2bcc76b32ff/tap7.md#exceptions
…--
vladimir.v.diaz@gmail.com
PGP fingerprint = ACCF 9DCA 73B9 862F 93C5 6608 63F8 90AA 1D25 3935
--
On Tue, May 16, 2017 at 5:32 PM, Lois Anne DeLong <notifications@github.com>
wrote:
I proofread TAP 7 again. I followed your instructions and continued working
on the "master." I committed and it opened a separate branch.
I think this is now in pretty good shape, except that it needs the example
scenarios and code. So, unless you make very large changes, you probably
won't need me on this anymore (so I can't mess up your git files).
Do let me know if and when you want me to look at TAP8 or 9.
Lois
On Mon, May 15, 2017 at 3:17 PM, Lois A DeLong ***@***.***> wrote:
> Ok. I'm sorry about that, Vlad.
>
> Lois
>
> On Mon, May 15, 2017 at 3:12 PM, Vladimir Diaz ***@***.***
>
> wrote:
>
>> It's not a problem, but it appears that you accidentally merged the TAP
7
>> pull request into the "master" branch here
>> <https://github.com/theupdateframework/taps/commit/
6abdb7a>,
>> and subsequent commits related to TAP 7 were then committed to the
"master"
>> branch:
>> https://github.com/theupdateframework/taps/commits/master
>>
>> I've changed it so that PRs cannot be accidentally merged until they've
>> been fully reviewed.
>>
>> You can continue making edits to TAP 7 on the "master" branch. We'll
just
>> update the status of the TAP 7 file once it's accepted.
>>
>> —
>> You are receiving this because you were mentioned.
>> Reply to this email directly, view it on GitHub
>> <https://github.com/theupdateframework/taps/pull/
19#issuecomment-301574623>,
>> or mute the thread
>> <https://github.com/notifications/unsubscribe-
auth/AN5qX-srcObDlAzL-p-VzLi-ufFyesGoks5r6KOkgaJpZM4M6PYc>
>> .
>>
>
>
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#19 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ADW5czk8UrttH4Pfd2hWArZBVqC7FRT_ks5r6hXWgaJpZM4M6PYc>
.
|
|
I thought you planned to add in some code illustrating what would happen in
certain situations (see the headings at the end). Have these examples just
been worked into other parts of the text? If so, let's get rid of the last
section that is now empty.
Lois
On Wed, May 17, 2017 at 11:28 AM, Vladimir Diaz <notifications@github.com>
wrote:
… Justin,
What do you think of the revised TAP 7?
I think this is now in pretty good shape, except that it needs the example
> scenarios and code.
I did add example scenarios and code. For instance, the modifying an
implementation to exit with expected return codes
<https://github.com/theupdateframework/taps/blob/
fd62820/tap7.md#modifying-
an-implementation-to-exit-with-expected-return-codes>
subsection is covered here:
https://github.com/theupdateframework/taps/blob/
fd62820/tap7.md#exceptions
--
***@***.***
PGP fingerprint = ACCF 9DCA 73B9 862F 93C5 6608 63F8 90AA 1D25 3935
--
On Tue, May 16, 2017 at 5:32 PM, Lois Anne DeLong <
***@***.***>
wrote:
> I proofread TAP 7 again. I followed your instructions and continued
working
> on the "master." I committed and it opened a separate branch.
>
> I think this is now in pretty good shape, except that it needs the
example
> scenarios and code. So, unless you make very large changes, you probably
> won't need me on this anymore (so I can't mess up your git files).
>
> Do let me know if and when you want me to look at TAP8 or 9.
>
> Lois
>
>
> On Mon, May 15, 2017 at 3:17 PM, Lois A DeLong ***@***.***> wrote:
>
> > Ok. I'm sorry about that, Vlad.
> >
> > Lois
> >
> > On Mon, May 15, 2017 at 3:12 PM, Vladimir Diaz <
***@***.***
> >
> > wrote:
> >
> >> It's not a problem, but it appears that you accidentally merged the
TAP
> 7
> >> pull request into the "master" branch here
> >> <https://github.com/theupdateframework/taps/commit/
> 6abdb7a>,
> >> and subsequent commits related to TAP 7 were then committed to the
> "master"
> >> branch:
> >> https://github.com/theupdateframework/taps/commits/master
> >>
> >> I've changed it so that PRs cannot be accidentally merged until
they've
> >> been fully reviewed.
> >>
> >> You can continue making edits to TAP 7 on the "master" branch. We'll
> just
> >> update the status of the TAP 7 file once it's accepted.
> >>
> >> —
> >> You are receiving this because you were mentioned.
> >> Reply to this email directly, view it on GitHub
> >> <https://github.com/theupdateframework/taps/pull/
> 19#issuecomment-301574623>,
> >> or mute the thread
> >> <https://github.com/notifications/unsubscribe-
> auth/AN5qX-srcObDlAzL-p-VzLi-ufFyesGoks5r6KOkgaJpZM4M6PYc>
> >> .
> >>
> >
> >
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/theupdateframework/taps/pull/
19#issuecomment-301921988>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/
ADW5czk8UrttH4Pfd2hWArZBVqC7FRT_ks5r6hXWgaJpZM4M6PYc>
> .
>
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#19 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AN5qX3Sy6u6X7ZdaS-4bXloP6mS9Yrmoks5r6xIfgaJpZM4M6PYc>
.
|
Conformance testing can determine whether an implementation meets the requirements set by a specification. A tool that helps developers and users test that an implementation bahaves according to the TUF specification does not presently exist. Although the reference implementation contains unit tests that verify correct behavior, such as updating metadata in the expected order and blocking known updater attacks, these unit tests only work with the reference implementation. Conformance testing should instead work across different languages and platforms. In other words, the specification should endorse an official tool, compatible with any implementation, and cover how an implementation can be set up for conformance testing.
The implementation of TAP 7 is a work in progress and is available for testing here