Skip to content

Commit

Permalink
v1.1 publication (#49)
Browse files Browse the repository at this point in the history
* two editorial fixes (#40)

* Added contribution from other repo

* working

* working

* working

* Update toolbox.adoc

Updated text, added specific number of PAIs to be created in the test (as opposed to having this in individual toolbox tests for consistency)

* Update toolbox.adoc

Editorial corrections

* Update toolbox.adoc

Updated the pass/fail criteria based on @The-Fiona comments

* Updates to Eye Toolbox

Added new test list (with sensor applicability).

Modified test sets to add number of species types to be produced (including all variants).

Removed "Other" section as this is now in main toolbox overview.

* added to toolbox.adoc

* Migrated vein to adoc

Created attack subfolder, moved all attacks there. Renamed "general.md" to "vein_PADtesting.adoc"

No content changes (other than changing Variations to "None")

* migrated face to adoc

Moved from TXT files to adoc format, not content change.

Moved files into attacks folder to fit with eye (and PR for vein)

* migrated remaining tests to adoc

I migrated the remaining docs to adoc. I copied the attack potential table to all the tests (without changes except in tests 5 becasue those are largely blank so I put in "?"). I moved the remaining files to make it follow the other toolbox updates.

I moved the inventory document to adoc as well.

* IND to FUN

Edited ATE_IND.1 to ATE_FUN.1

* Update toolbox.adoc

undo the ATE_IND to ATE_FUN back to ATE_IND

* remove TODO line

Remove the TODO line based on discussion on 5/16 to leave it as transactions for vendor flexibility

* Single subject update

Based on the 5/30 call this proposed change is to specify the number of subjects to be used in the creation of PAI.

* changed date

changed date

* fixed internal reference link

Similar problem with reference links as in PP-Module

* Clarification about AVA_VAN

Based on comments from Mary Baisch, this update attempts to clarify that that toolbox is not intended for AVA_VAN but can be used as the basis for those tests with modifications that the evaluator may decide to make.

Also found a misspelling, and added the :icons: font in the header (though I removed the section that specifically needed it).

* Revert "Clarification about AVA_VAN"

This reverts commit 05e5e51.

* Initial update for PR1

Based on comments by Mary Baisch this is intended to further clarify how the toolbox could be used as a basis for the creation of AVA_VAN tests while not being written for that purpose.

Also fixed a misspelling and added :icons: font line to the header.

* Toolbox Template Example

This is in response to #18 about related to having a standardized template for a toolbox.

There are 4 files in the main folder (here BIO is used to mark the modality, so this should be EYE, FACE, FINGER, etc):

PAD Testing - the description of anything specific to this modality that needs to be documented (in addition to the toolbox overview) along with an introduction to the toolbox.

List - a table list of all the tests and any applicability notes (like these are only relevant with certain types of sensors or other considerations)

Inventory - a list of the inventory that is used in the test, things like paper, printers, camera, the things used to make the PAI

References - external references for any of the attacks (a master list)

Within the main folder there would then be subfoldlers marked with XX_<attack category>_attacks. The XX would be numbering for increasing difficulty (i.e. the lowest level PAI, simplest to create would be 01, the most difficult test would be the top number). The <attack category> would be some sort of title that would provide some clarity as to what the tests will be used as a source for the PAI.

Within each folder then, you would have the files names XX_YY_attack where the XX matches the folder number and YY is the test number.

The List table should match out all the numbers. While I don't have it in here, there is a (Vx) listing for some of the tests, this is when there is a number of variants for the specific test available (so a test with a V1 and V2 would have 2 variants in addition to the "base" test). This could be handled in another way though.

This used the eye tests to fill out the template, but I didn't modify any of the files.

* Revert "Toolbox Template Example"

This reverts commit 8a48e39.

* Revert "Revert "Toolbox Template Example""

This reverts commit aba98fa.

* Revert "Revert "Revert "Toolbox Template Example"""

This reverts commit 6e2d608.

* Revert "Revert "Revert "Revert "Toolbox Template Example""""

This reverts commit 448dbb8.

* Revert "Revert "Revert "Revert "Revert "Toolbox Template Example"""""

This reverts commit 645dcfa.

* Toolbox Template Example

This is in response to #18 about related to having a standardized template for a toolbox.

There are 4 files in the main folder (here BIO is used to mark the modality, so this should be EYE, FACE, FINGER, etc):

PAD Testing - the description of anything specific to this modality that needs to be documented (in addition to the toolbox overview) along with an introduction to the toolbox.

List - a table list of all the tests and any applicability notes (like these are only relevant with certain types of sensors or other considerations)

Inventory - a list of the inventory that is used in the test, things like paper, printers, camera, the things used to make the PAI

References - external references for any of the attacks (a master list)

Within the main folder there would then be subfoldlers marked with XX_<attack category>_attacks. The XX would be numbering for increasing difficulty (i.e. the lowest level PAI, simplest to create would be 01, the most difficult test would be the top number). The <attack category> would be some sort of title that would provide some clarity as to what the tests will be used as a source for the PAI.

Within each folder then, you would have the files names XX_YY_attack where the XX matches the folder number and YY is the test number.

The List table should match out all the numbers. While I don't have it in here, there is a (Vx) listing for some of the tests, this is when there is a number of variants for the specific test available (so a test with a V1 and V2 would have 2 variants in addition to the "base" test). This could be handled in another way though.

This used the eye tests to fill out the template, but I didn't modify any of the files.

* update to 3 subjects

Update to three subjects as noted in #17

* Updated to match new toolbox template

This is an update to match the new toolbox template. No changes were made to the attacks themselves. Previous attack set 01 has been removed as the scan and reprint tests have been removed from Face as well.

* Eye Attack Potential update

Added Attack Potential tables for each attack and each species type for each attack. All calculations show under the 13 limit.

* Update for clarification

This is to close #22 as agreed on the 12/5 call.

* Test procedure update

This is to close #23 related to clarifying the overall testing description.

The table was also updated to follow a more "standard" format.

The PP name was updated to match the current.

* first commit

* Update toolbox.adoc

* Update toolbox.adoc

* Update toolbox.adoc

* Update toolbox.adoc

* Brian toolbox edits

My suggested edits to the changes.

I have edited some text (moved a few things around), did some grammar checking, and edited the tables.

* editorial update by Kai

I updated the overview to fix some inconsistency or mistakes

* research paper authority

One line update to strengthen the use of research papers

* 1st commit

* 2nd commit

* 3rd commit

* 1st commit

* Update and rename 2D-face_attack_1-2.adoc to 2D-face_attack_1_1.adoc

* Update 2D_Face_Toolbox_overview.adoc

* Update 2D-face_attack_1_1.adoc

* Update 2D_Face_Toolbox_Tool_Inventory.adoc

* Update and rename references.adoc to 2D_Face_Toolbox_References.adoc

* Update 2D-face_attack_1_1.adoc

* Update and rename 2D-face_attack_1-3.adoc to 2D-face_attack_1_2.adoc

* Update 2D-face_attack_1_2.adoc

* Update 2D-face_attack_1_1.adoc

* Update and rename 2D-face_attack_2-1.adoc to 2D-face_attack_1_3.adoc

* Update 2D-face_attack_1_1.adoc

* Update 2D-face_attack_1_2.adoc

* Update 2D-face_attack_1_1.adoc

* Update 2D-face_attack_1_2.adoc

* Update 2D-face_attack_1_2.adoc

* Update 2D-face_attack_1_3.adoc

* Update and rename 2D-face_attack_2-2.adoc to 2D-face_attack_2-1.adoc

* Delete attack_1-1.adoc

* Update and rename 2D-face_attack_2-3.adoc to 2D-face_attack_2-2.adoc

* Update and rename 2D-face_attack_2-4.adoc to 2D-face_attack_2-3.adoc

* Delete 2D-face_attack_3-1.adoc

* Delete 2D-face_attack_3-2.adoc

* Delete 2D-face_attack_3-3.adoc

* Delete 2D-face_attack_4-1.adoc

* Delete 2D-face_attack_4-2.adoc

* Delete 2D-face_attack_5-1.adoc

* Delete 2D-face_attack_5-2.adoc

* Update 2D_Face_Toolbox_overview.adoc

* Update 2D_Face_Toolbox_Tool_Inventory.adoc

* Update 2D_Face_Toolbox_Tool_Inventory.adoc

* Rename 2D-face_attack_2-1.adoc to 2D-face_attack_2_1.adoc

* Rename 2D-face_attack_2-2.adoc to 2D-face_attack_2_2.adoc

* Rename 2D-face_attack_2-3.adoc to 2D-face_attack_2_3.adoc

* Create 2D_Face_Verification_List.adoc

* Updated based on NK comments

* Update 2D-face_attack_1_3.adoc

* Update 2D-face_attack_1_1.adoc

* Update 2D-face_attack_1_2.adoc

* Update 2D_Face_Verification_List.adoc

* Delete 2D-face_attack_2_1.adoc

* Delete 2D-face_attack_2_2.adoc

* Delete 2D-face_attack_2_3.adoc

* Update toolbox.adoc

* Update toolbox.adoc

* Update toolbox overview

Change the text as suggested by @woodbe in  #32

* Update 2D-face_attack_1_1.adoc

* Update 2D-face_attack_1_2.adoc

* Update 2D-face_attack_1_3.adoc

* editorial fixes

* 3D face toolbox overview

* move file

* Delete 3D_Face_Toolbox_overview.adoc

* editorial fixes

* Update 3D_Face_Toolbox_overview.adoc

* final update

* editorial update

* editorial update

* Update 2D-face_attack_1_1.adoc

* editorial update

* Create 3D-face_attack_1_1.adoc

* Update 2D_Face_Toolbox_Tool_Inventory.adoc

* Create 3D_Face_Toolbox_Tool_Inventory.adoc

* Create 3D-face_attack_3.adoc

* Create 3D_Face_Verification_List.adoc

* Create 3D_Face_Toolbox_References.adoc

* Updates from Brian

I made several updates here. Most focused on language in the reading to be a little clearer. Some table adjustments were also made to present the tables a little better.

* Update 3D_Face_Toolbox_Tool_Inventory.adoc

* Editorial update for proposed release

This is editorial related to the biometricITC/cPP-biometrics#266 to try to polish the Toolbox for release at the same time.

I added a revision history and changed SD to BIOSD. I also edited the title to match with the rest of the docs.

* Update toolbox.adoc

update based on @n-kai comment to move this to the Toolbox from the SD.

* Removal of old toolboxes

Current face and eye toolboxes have been moved to their own repositories, so these are being deleted.

* Removal of old files

These files are being removed as out of date

* Deleting template for toolbox

This template is now out of date. A new one may need to be created, but probably not right now.

* Update toolbox.adoc

Updates based on MITRE's review.

* Updates for public release

These are focused on updating the dates and versions for all the documents in preparation of public release on May 11, 2020.

* Update toolbox.adoc

Added doctype for proper formatting of PDF output

* Move of vein to own repository

moved all vein to separate repository

* two editorial fixes

Fix the errors in table

Co-authored-by: nils <nt@konfidas.de>
Co-authored-by: Brian Wood <be.wood@samsung.com>
Co-authored-by: The-Fiona <37903201+The-Fiona@users.noreply.github.com>
Co-authored-by: ccolin318 <56977088+ccolin318@users.noreply.github.com>

* Update toolbox.adoc (#42)

* Update toolbox.adoc (#44)

* update pass fail criteria and editorial fixes (#47)

* update pass fail criteria and editorial fixes

* Update toolbox.adoc

Co-authored-by: Greg Fiumara <gregory.fiumara@nist.gov>

* Update toolbox.adoc

Co-authored-by: Greg Fiumara <gregory.fiumara@nist.gov>

* Update toolbox.adoc

Co-authored-by: Greg Fiumara <gregory.fiumara@nist.gov>

Co-authored-by: Brian Wood <woodbe@google.com>
Co-authored-by: Greg Fiumara <gregory.fiumara@nist.gov>

* Changes for publication

Summary of changes:

- version number to v1.1
- date changes

* Revert "Changes for publication"

This reverts commit e7730db.

* Changes for publication

Summary of changes:

- version number to v1.1
- date changes

Co-authored-by: n-kai <n-kai@ipa.go.jp>
Co-authored-by: nils <nt@konfidas.de>
Co-authored-by: Brian Wood <be.wood@samsung.com>
Co-authored-by: The-Fiona <37903201+The-Fiona@users.noreply.github.com>
Co-authored-by: ccolin318 <56977088+ccolin318@users.noreply.github.com>
Co-authored-by: Greg Fiumara <gregory.fiumara@nist.gov>
  • Loading branch information
7 people committed Nov 12, 2021
1 parent cffb2f7 commit d7acd65
Showing 1 changed file with 29 additions and 24 deletions.
53 changes: 29 additions & 24 deletions toolbox.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,15 @@
:sectnums:
:imagesdir: images
:icons: font
:revnumber: 1.0
:revdate: May 11, 2020
:revnumber: 1.1
:revdate: November 9, 2021
:doctype: book
:xrefstyle: full

== Introduction
The TOE may be vulnerable to presentation attacks where attackers attempt to subvert the biometric enrolment or verification by presenting the Presentation Attack Instruments (PAIs). There is a wide range of PAIs that can be used, including natural biometric characteristics, such as dead eyes, or artefacts created from copied or faked characteristics. Using natural biometric characteristics is out of scope of <<BIOPP-Module>> evaluation and the evaluator shall only use created artefacts to evaluate the TOE.

The toolbox defines the common artefacts for each biometric modality based on publicly available information (e.g. research papers), experiences and knowledge shared among the BIO-iTC members. The evaluator needs to read the <<BIOSD>> Section 6 as it explains how the evaluator shall use the toolbox during the ATE_IND.1 (Independent testing) and AVA_VAN.1 (Penetration testing) evaluation for Presentation Attack Detection (PAD) in detail.
The toolbox defines the common artefacts for each biometric modality based on publicly available information (e.g. research papers), experiences and knowledge shared among the BIO-iTC members. The evaluator needs to read the <<BIOSD>> Section 7 as it explains how the evaluator shall use the toolbox during the ATE_IND.1 (Independent testing) and AVA_VAN.1 (Penetration testing) evaluation for Presentation Attack Detection (PAD) in detail.

This overview is originally developed for evaluation activities for FIA_MBV_EXT.3, however, the evaluator can apply the same principles to evaluation activities for FIA_MBE_EXT.3.

Expand Down Expand Up @@ -40,16 +41,16 @@ _modality name_ Toolbox overview::
This section provides specific information only applicable to relevant biometric modality.

_modality name_ Toolbox Inventory::
This section categorizes tools and materials that the evaluator shall use to capture a image of biometric characteristics and produce artefacts.
This section categorizes tools and materials that the evaluator shall use to capture an image of biometric characteristics and produce artefacts.

_modality name_ Verification List::
This section summarizes all test items that the evaluator shall peform during independent testing. As explained in <<BIOSD>> Section 6, the evaluator shall select specific test items for penetration testing based on the result of independent testing.
This section summarizes all test items that the evaluator shall perform during independent testing. As explained in <<BIOSD>> Section 7, the evaluator shall select specific test items for penetration testing based on the result of independent testing.

_modality name_ References::
This section lists all publicly available information referred to create a toolbox.

Test items::
Each test item includes the following sub-sections. This toolbox overview provide a general test protocol in common for all toolboxes and these test items describe more detailed information to enable repeatable testing.
Each test item includes the following sub-sections. This toolbox overview provides a general test protocol in common for all toolboxes and these test items describe more detailed information to enable repeatable testing.
+
[cols=".^1,2",options="header"]
|===
Expand All @@ -69,14 +70,14 @@ Each test item includes the following sub-sections. This toolbox overview provid
|*Input*
|Required input to produce artefacts

|*Tools*
|Required tools to capture a image of biometric characteristics and produce artefacts
|*Attack Tools/Media*
|Required tools and media to capture an image of biometric characteristics and produce artefacts

|*Recipe*
|Procedure to create artefacts

|*Variations*
|Variants of artefacts to be generated based on this test item. The evaluator shall create those variants by slightly different *Recipe* or with different *Tools* specified here for the independent testing.
|Variants of artefacts to be generated based on this test item. The evaluator shall create those variants by slightly different procedure (e.g. different *Recipe* or with different *Attack Tools/Media* specified here) for the independent testing.

|*Prerequisite*
|Any conditions that should meet to perform each test
Expand All @@ -88,12 +89,12 @@ Each test item includes the following sub-sections. This toolbox overview provid
|Suggestions that the evaluator should consider devising penetration tests from this test item and calculate the attack potential rating. The evaluator may change the rating considering actual expertise or knowledge of TOE used to succeed attacks, however, the evaluator shall report such changes with proper justification

|*Pass Criteria*
|If this Pass criteria is defined in test items, evaluator shall follow it. Otherwise, the evaluator shall follow criteria defined in this toolbox overview for independent testing and one defined in <<BIOSD>> Section 9.3 for penetration testing
|Additional information for pass-fail criteria of the test (IAPAR shall not exceed the value assigned in FIA_MBV_EXT.3.1 in any case)

|===

== Common guidance for Independent & Vulnerability Testing
As explained in <<BIOPP-Module>>, the TOE is the whole biometric system, including Comparison, Decision and Presentation Attack Detection Subsystems. This means in order to successfully overcome the TOE by the use of artefacts, a genuine person (test subject) has to be enroled into the TOE, artefacts have to be created referring the toolbox for the corresponding biometric modality and artefacts have to produce a attack presentation match (i.e. a successful presentation attack).
As explained in <<BIOPP-Module>>, the TOE is the whole biometric system, including Comparison, Decision and Presentation Attack Detection Subsystems. This means in order to successfully overcome the TOE by the use of artefacts, a genuine person (test subject) has to be enroled into the TOE, artefacts have to be created referring the toolbox for the corresponding biometric modality and artefacts have to produce an attack presentation match (i.e. a successful presentation attack).

For all types of testing, there are some common steps/procedures to be followed. These are detailed here.

Expand Down Expand Up @@ -143,7 +144,7 @@ For independent testing, this guidance is common for all toolboxes. More specifi
This is in addition to guidance in <<Common guidance for Independent & Vulnerability Testing>>.

=== General test protocol
The presentation attack can be performed through the following two steps after performing Preparation in Section 5.
The presentation attack can be performed through the following two steps after performing <<Preparation>>.

==== Artefact production
The production of artefacts for each toolbox shall be performed as follows:
Expand All @@ -163,8 +164,6 @@ The evaluator shall present artefacts to the TOE to perform presentation attacks
The evaluator shall prepare three test subjects for the above test. A test subject is defined as one individual, and not different body parts from one person (i.e. three fingers from one person could not be considered to be three test subjects for the creation of artefacts).

=== Pass/Fail Criteria
The following pass criteria shall be applied if no other criteria are defined in the toolbox.

A TOE passes the test if and only if it reliably defeats the use of *all artefacts (i.e. 3 X 3 = 9 artefacts in total)* that have to be built according to the toolbox. This means that none of the artefacts must be able to reproducibly overcome the TOE.

To reproducibly overcome the TOE by the use of a *certain artefact* in the outlined test scenario is defined as follows:
Expand All @@ -173,7 +172,7 @@ To reproducibly overcome the TOE by the use of a *certain artefact* in the outli
[cols="1,1,3",options="header"]
|===

|Attempts
|Attempts/Transactions
|Number of matches
|Outcome

Expand All @@ -187,7 +186,7 @@ To reproducibly overcome the TOE by the use of a *certain artefact* in the outli

|10
|2
|Additional ten (10) attempts shall be made
|Additional ten (10) attempts/transactions shall be made

|20
|2
Expand All @@ -199,22 +198,24 @@ To reproducibly overcome the TOE by the use of a *certain artefact* in the outli

|===

The maximum number of attempts allowed with one artefact is twenty (20). If three (3) matches are made to the artefact, the independent test fails (further attempts are not necessary even if 20 total attempts have not yet been made).
As explained in <<BIOPP-Module>>, presentation attacks can be done by attempts or transactions.

The maximum number of attempts/transactions allowed with one artefact is twenty (20). If three (3) matches are made to the artefact, the independent test fails (further attempts/transactions are not necessary even if 20 total attempts/transactions have not yet been made) because the IAPAR has exceeded 15%, the allowable maximum value specified in FIA_MBV_EXT.3.1.

== Guidance for Penetration Testing (AVA_VAN.1)
The evaluator moves to penetration testing only if the TOE passes independent testing. As described in <<BIOSD>> Section 6, the evaluator shall select those artefacts that show higher imposter attack presentation match rate during independent testing or higher quality artefacts.
The evaluator moves to penetration testing only if the TOE passes independent testing. As described in <<BIOSD>> Section 7, the evaluator shall select those artefacts that show a higher IAPAR during independent testing or higher quality artefacts.

This is in addition to guidance in <<Common guidance for Independent & Vulnerability Testing>>.

=== General test protocol
Presentation attack can be performed through the following two steps after performing Preparation in Section 5.
Presentation attack can be performed through the following two steps after performing <<Preparation>>.

==== Artefact production
The production of artefacts for each toolbox shall be performed as follows:

* The evaluator should select artefacts in a toolbox that may produce attack presentation match at higher probability considering the result of independent testing.

* The evaluator may refine the production process of artefacts, as explained in <<BIOSD>> Section 6. The toolbox describes generalized process to produce artefacts referring to research papers. These research papers may describe more detailed information to produce better artefacts. Such information is valuable if the TOE's PAD algorithm is the same or similar to ones tested by researchers. The evaluator shall consider relevant research papers to be authoritative over the generalized descriptions provided in a toolbox for improving the creation of artefacts.
* The evaluator may refine the production process of artefacts, as explained in <<BIOSD>> Section 7. The toolbox describes generalized process to produce artefacts referring to research papers. These research papers may describe more detailed information to produce better artefacts. Such information is valuable if the TOE's PAD algorithm is the same or similar to ones tested by researchers. The evaluator shall consider relevant research papers to be authoritative over the generalized descriptions provided in a toolbox for improving the creation of artefacts.

* The evaluator may produce an arbitrary number of artefacts from each test subject within allowed time period. As described in <<BIOSD>>, both independent and penetration testing shall be finished within one week.

Expand All @@ -224,15 +225,15 @@ The evaluator shall present artefacts to the TOE to perform presentation attacks
* Each artefact shall be presented to the TOE an arbitrary number of times within allowed time period. As described in <<BIOSD>>, both independent and penetration testing shall be finished within one week.

=== Number of Subjects
If the evaluator can create artefacts that produce an attack presentation match during independent testing, the evaluator should select the test subjects whose artefacts had successful matches and increase the number of attempts. The evaluator may replace the test subject for penetration testing as described in <<BIOSD>> Section 6.
If the evaluator can create artefacts that produce an attack presentation match during independent testing, the evaluator should select the test subjects whose artefacts had successful matches and increase the number of attempts/transactions. The evaluator may replace the test subject for penetration testing as described in <<BIOSD>> Section 7.

=== Pass/Fail Criteria
As described in <<BIOSD>>, both independent and penetration testing shall be finished within one week. The evaluator may select one or two artefacts and perform an arbitrary number of attempts within this time period. If the evaluator can create artefacts that meet the criteria defined in <<BIOSD>> Section 9.3, the TOE fails AVA_VAN.1 evaluation.
As described in <<BIOSD>>, both independent and penetration testing shall be finished within one week. The evaluator may select one or two artefacts and perform an arbitrary number of attempts/transactions within this time period. If the evaluator can create artefacts that reproducibly cause the TOE to achieve an IAPAR higher than what is specified in FIA_MBV_EXT.3.1, the TOE fails AVA_VAN.1 evaluation.

== Related Documents

- [#BIOPP-Module]#[BIOPP-Module]# collaborative PP-Module for Biometric enrolment and verification - for unlocking the device -, May 11, 2020, Version 1.0
- [#BIOSD]#[BIOSD]# Supporting Document Mandatory Technical Document: Evaluation Activities for collaborative PP-Module for Biometric enrolment and verification - for unlocking the device -, May 11, 2020, Version 1.0
- [#BIOPP-Module]#[BIOPP-Module]# collaborative PP-Module for Biometric enrolment and verification - for unlocking the device -, November 9, 2021, Version 1.1
- [#BIOSD]#[BIOSD]# Supporting Document Mandatory Technical Document: Evaluation Activities for collaborative PP-Module for Biometric enrolment and verification - for unlocking the device -, November 9, 2021, Version 1.1

== Revision History

Expand All @@ -259,4 +260,8 @@ As described in <<BIOSD>>, both independent and penetration testing shall be fin
|May 11, 2020
|Public Release

|1.1
|November 9, 2021
|Update based on changes to the PP-Module v1.1

|===

0 comments on commit d7acd65

Please sign in to comment.