Implement approvals in IGSDB

[christoph-maurer]

Approvals are a cryptographic signature which states that the data sets fulfills certain requirements. They will be used by software applications to filter suitable high-quality data sets.

Currently we think about approvals for NFRC and AERC and possibly for LBNL research level and generic records.

[christoph-maurer]

ISE will prepare a proposal.

[danielmcquillen]

@christoph-maurer Where does the proposal for this feature stand? Is it related to the issue that mentions hashes?

[RDmitchell]

@christoph-maurer -- in order to understand what this issue entails, we would need a detailed proposal/spec.

[christoph-maurer]

@RDmitchell @danielmcquillen The approvals are defined in https://github.com/building-envelope-data/api/blob/develop/apis/database.graphql#L1507-L1794 . @StephenCzarnecki and @simon-wacker had the discussion #187 which may be helpful. We have set the priority of this issue to low because the synchronization of the uuids of components and institutions has a much higher priority for the software companies.

[simon-wacker]

@danielmcquillen Currently the specification of approvals in https://github.com/building-envelope-data/api/blob/develop/apis/database.graphql#L1507-L1794 needs some polishing. I started working on it but then got held up by https://github.com/LBNL-ETA/igsdb-webapp/pull/229 I'll continue with the polishing / fine-tuning / updating the specs as soon as possible.

[simon-wacker]

@danielmcquillen @RDmitchell At least for reference: I polished the specification for approvals in the API documentation:

api/apis/database.graphql

Lines 1681 to 2202 in dc15b3c

	"""
	Approval with
	[GnuPG](https://www.gnupg.org)
	signature.
	To verify an approval,
	1. install [GnuPG](https://www.gnupg.org),
	2. save the approved message in the file `./message.json` (it is the value of
	the field `message`),
	3. save the ASCII-armored signature in the file `./message.json.asc` (it is the
	value of the field `signature`),
	4. * receive (and import) the public key for the given fingerprint from the
	[OpenPGP Keyserver](https://keys.openpgp.org) by running
	`gpg --keyserver hkps://keys.openpgp.org/ --receive-keys <KEY_FINGERPRINT>`
	where `<KEY_FINGERPRINT>` is the value of the field `keyFingerprint`.
	* make sure that the key was not expired and was not revoked when the
	approval was created by checking that the output of
	`gpg --list-keys <KEY_FINGERPRINT>`
	does not say so: Look for the terms 'revoked' and 'expired'.
	* make sure that the key belongs to the approver and was allowed for signing
	and not revoked when the approval was created by sending a request of the
	form
	```
	query {
	institution(id: <APPROVER_ID>) {
	hasGnuPgKeyFingerprint(
	where: {
	and: [
	{ fingerprint: { equalTo: <KEY_FINGERPRINT> } }
	{ allowedAt: { lessThanOrEqualTo: <TIMESTAMP> } }
	{ forbiddenAt: { greaterThan: <TIMESTAMP> } }
	]
	}
	)
	}
	}
	```
	to the metabase and checking that the response is
	```
	{
	"data": {
	"institution": {
	"hasGnuPgKeyFingerprint": true
	}
	}
	}
	```
	where `<APPROVER_ID>`, `<KEY_FINGERPRINT>`, and `<TIMESTAMP>` are the values
	of the fields `approverId`, `keyFingerprint`, and `timestamp`. Alternatively,
	retrieve the allowed-for-signing and not-revoked-at-the-given-time key
	fingerprint from the metabase's GraphQL endpoint by sending a request of
	the form
	```
	query {
	institution(id: <APPROVER_ID>) {
	gnuPgKeyFingerprints(
	where: {
	and: [
	{ fingerprint: { equalTo: <KEY_FINGERPRINT> } }
	{ allowedAt: { lessThanOrEqualTo: <TIMESTAMP> } }
	{ forbiddenAt: { greaterThan: <TIMESTAMP> } }
	]
	}
	) {
	edges {
	node {
	fingerprint
	allowedAt
	forbiddenAt
	user {
	node {
	uuid
	name
	}
	}
	}
	}
	totalCount
	}
	}
	}
	```
	to the metabase and check that the response
	```
	{
	"data": {
	"institution": {
	"gnuPgKeyFingerprints": {
	"edges": [
	{
	"node": {
	"fingerprint": <KEY_FINGERPRINT>,
	"allowedAt": ...,
	"forbiddenAt": ...,
	"user": {
	"node": {
	"uuid": ...,
	"name": ...
	}
	}
	}
	}
	],
	"totalCount": 1
	}
	}
	}
	}
	```
	contains exactly one edge (and node).
	5. verify the signature with the public key by running
	`gpg --verify ./message.json.asc ./message.json`.
	To create an approval,
	1. install [GnuPG](https://www.gnupg.org),
	2. create a master key by running
	`gpg --full-generate-key`,
	selecting option `3. DSA (sign only)`,
	choosing the keysize `2048`,
	setting the validity `2y = key expires in 2 years`, and
	answering all subsequent questions in particular adding a strong passphrase.
	3. create a signing subkey by running
	`gpg --edit-key <YOUR_EMAIL_ADDRESS>`,
	entering the command `addkey`
	selecting option `(3) DSA (sign only)`,
	choosing the keysize `2048`,
	setting the validity `2y = key expires in 2 years`, and
	answering all subsequent questions, and
	entering the command `save`.
	4. list your keys with fingerprints by running
	`gpg --list-keys --with-subkey-fingerprint <YOUR_EMAIL_ADDRESS>`.
	5. add your signing key to the
	[OpenPGP Keyserver](https://keys.openpgp.org)
	by running
	`gpg --keyserver hkps://keys.openpgp.org --send-keys <SIGNING_KEY_FINGERPRINT>`
	or by exporting the public key by running
	`gpg --export --armor --output ./signing_key.pub.asc <SIGNING_KEY_FINGERPRINT>`
	and uploading it via the form on
	[OpenPGP Key Upload](https://keys.openpgp.org/upload).
	6. within a few minutes, verify the signing key's user ID, which is
	`<YOUR_EMAIL_ADDRESS>`, by following the verification link sent to you.
	7. add the signing key fingerprint to the metabase with the GraphQL mutation
	```
	mutation {
	addGnuPgKeyFingerprint(input: {
	institutionId: <APPROVER_ID>
	fingerprint: <SIGNING_KEY_FINGERPRINT>
	}) {
	errors {
	code
	message
	path
	}
	}
	}
	```
	and ask the institution owner to allow the key for signing with the GraphQL
	mutation
	```
	mutation {
	allowGnuPgKeyFingerprint(input: {
	fingerprint: <SIGNING_KEY_FINGERPRINT>
	}) {
	errors {
	code
	message
	path
	}
	}
	}
	```
	Note that if you are the owner, the key is allowed automatically.
	For a response approval,
	8. save the response of the present query in a file `./message.json`.
	9. compute the signature and write it to `./message.json.asc` by running
	`gpg --detach-sig --armor --local-user <SIGNING_KEY_FINGERPRINT> --output ./message.json.asc ./message.json`, and
	10. add the approval to the final response of the present query through the
	field `approval`.
	For a data approval,
	8. query meta information of the data set to be signed with a GraphQL query
	like
	```
	query {
	opticalData(id: <OPTICAL_DATA_ID>) {
	uuid
	name
	description
	resourceTree {
	root {
	value {
	hashValue
	locator
	dataFormatId
	}
	}
	}
	}
	}
	```
	and load the data with an HTTPS GET request to the locator URL by running
	`wget <LOCATOR>`.
	9. review whether some statement is true for the data, for example, whether the
	data conforms to some standard. If that is so, query the meta information to
	be signed which at least contains the hash value of the data
	```
	query {
	opticalData(id: <OPTICAL_DATA_ID>) {
	resourceTree {
	root {
	value {
	hashValue
	}
	}
	}
	}
	}
	```
	make sure that the hash value is the same as before, wrap the statement and
	the meta information (named `<RESPONSE>` below) like so
	```
	{
	"statement": <STATEMENT>,
	"response": <RESPONSE>
	}
	```
	and save this in a file `./message.json`. The query should ask for the
	information necessary to identify the meta information and the data set
	itself uniquely like its ID and the data set's fingerprint, aka, hash value.
	It should not ask for information whose change would invalidate the approval
	unnecessaritly like the the verbal description of the data set.
	10. compute the signature and write it to `./message.json.asc` by running
	`gpg --detach-sig --armor --local-user <SIGNING_KEY_FINGERPRINT> --output ./message.json.asc ./message.json`, and
	11. add the signature as data approval to the database through its interface,
	which could for example have a GraphQL mutation `AddDataApproval`.
	Best practices for managing GnuPG keys within an institution:
	* In the metabase, each institution has at least one representative. The
	institution's GnuPG keys are the ones whose fingerprints are associated with
	the institution. Fingerprints can be added by representatives and allowed and
	forbidden for signing by instiution owners. The corresponding GraphQL
	mutations are `addGnuPgKeyFingerprint`, `allowGnuPgKeyFingerprint`, and
	`forbidGnuPgKeyFingerprint`.
	* Each representative of an institution within the metabase who is supposed to
	approve data for its institution adds its signing key to the
	[OpenPGP Keyserver](https://keys.openpgp.org),
	the key fingerprint to the metabase, and lets the fingerprint be allowed for
	signing by the institution owner.
	* When a representative's signing subkey is compromised she revokes it as
	described in
	[Revoking key components](https://www.gnupg.org/gph/en/manual.html#AEN305),
	publishes the revocation to the keyserver by running
	`gpg --keyserver hkps://keys.openpgp.org --send-key <SIGNING_KEY_FINGERPRINT>`,
	and forbids the key for signing on the metabase with the mutation
	`forbidGnuPgKeyFingerprint`
	* When a representative leaves a company or is no longer allowed to approve data,
	the metabase is informed about that with the GraphQL mutation
	```
	mutation {
	removeInstitutionRepresentative(input: {
	userId: <USER_ID>
	institutionId: <INSTITUTION_ID>
	}) {
	errors {
	code
	message
	path
	}
	}
	}
	```
	which also forbids all GnuPG keys associated with this representative from
	signing.
	Best practices for accepting data approvals by databases:
	* Check that at the time of approval creation, the fingerprint belonged to
	a non-revoked and non-disabled GnuPG key.
	* Check that the key was allowed to make signatures for that institution.
	* Check that the query contains all necessary fields and sub-fields.
	* Check that the message is constructed properly according to the documentation
	of the implementing type `DataApproval`.
	* Check that the signature is valid for the given message.
	Note that GnuPG is compliant with
	[OpenPGP Message Format](https://tools.ietf.org/html/rfc4880).
	For an introduction to GnuPG consult
	[The GNU Privacy Handbook](https://www.gnupg.org/gph/en/manual.html)
	or the article series
	[Getting started with GPG (GnuPG)](https://www.redhat.com/sysadmin/getting-started-gpg),
	[How to create GPG keypairs](https://www.redhat.com/sysadmin/creating-gpg-keypairs), and
	[Digital signatures with GnuPG](https://www.redhat.com/sysadmin/digital-signatures-gnupg).
	"""
	interface Approval {
	"""
	Creation timestamp
	"""
	timestamp: DateTime!
	"""
	Detached ASCII-armored GnuPG signature of the message which includes the
	GraphQL query response.
	With the message saved in the file `./message.json`, the signature can
	be computed and written to `./message.json.asc` by running
	`gpg --detach-sig --armor --output ./message.json.asc ./message.json`
	(to write it to standard output instead, use the option `--output -`). And,
	the signature can be verified by running
	`gpg --verify ./message.json.asc ./message.json`.
	Note that to compute a signature you need a GnuPG signing private key in your
	key ring and to verify a signature you need the GnuPG signing public key
	whose corresponding private key was used to compute the signature.
	"""
	signature: String!
	"""
	Fingerprint of the GnuPG signing key used to compute the signature.
	The key's fingerprint, long identifier, and short identifier are related as
	in the following example:
	* fingerprint: 0D69 E11F 12BD BA07 7B37 26AB 4E1F 799A A4FF 2279
	* long id: 4E1F 799A A4FF 2279
	* short id: A4FF 2279
	Fingerprints can be used anywhere where an identifier is expected. For
	security reasons it is used instead of long and short identifier. Its only
	disadvantage is that it is less practical for manual human use.
	"""
	keyFingerprint: String!
	"""
	[GraphQL query](https://graphql.org/learn/queries/)
	whose response is included in the message being signed, for example,
	```
	query OpticalData($id: Uuid!) {
	opticalData(id: $id) { ... }
	}
	```
	Specifying the query explicitely is necessary because approvals shall not
	become invalid when the GraphQL schema changes.
	"""
	query: GraphQlQuery!
	"""
	[GraphQL variables](https://graphql.org/learn/queries/#variables)
	defined by the GraphQL query, for example,
	```
	{ "id": "018731a4-4c0a-49af-a5d9-f377fbd0a939" }
	```
	"""
	variables: Any!
	"""
	Message that is signed.
	For `ResponseApproval` it is the serialized GraphQL response of `query`. And
	for `DataApproval` it is a serialized JSON combining the statement being made
	and the response. See the documentation of `ResponseApproval#message` and
	`DataApproval#message` for details.
	With the message put in the file `./message.json`, the GnuPG signature put in
	the file `./message.json.asc`, and the GnuPG public key of the approver added
	to the GnuPG key ring, the signature can be verified by running
	`gpg --verify ./message.json.asc ./message.json`.
	To compare the message `./message.json` and another message `./another.json`,
	run the command
	```
	diff --side-by-side --suppress-common-lines \
	<(jq --sort-keys --monochrome-output . ./message.json) \
	<(jq --sort-keys --monochrome-output . ./another.json)
	```
	in a
	[Bourne-again shell (Bash)](https://www.gnu.org/software/bash/),
	see
	[Faster and simpler with the command line: deep-comparing two 5GB JSON files 3X faster by ditching the code](https://genius.engineering/faster-and-simpler-with-the-command-line-deep-comparing-two-5gb-json-files-3x-faster-by-ditching-the-code/)
	and
	[Using jq or alternative command line tools to compare JSON files](https://stackoverflow.com/questions/31930041/using-jq-or-alternative-command-line-tools-to-compare-json-files/37175540#37175540).
	Comparing messages is necessary to verify that the data `./another.json` you
	received in a query that includes this approval matches the approved data.
	Which is of course the case if all involved organizations are trustworthy.
	However, verified signatures and matching messages together prove data
	integrity (at least if private keys used to make the signatures in the first
	place are not compromised).
	"""
	message: String!
	"""
	Approver identifier issued by the metabase. The approver's meta data can be
	obtained by sending the GraphQL query
	`query { institution(id: ...) { ... } }`
	to the metabase. For data approvals, the approver makes the statement about
	the data identified by the query. For response approvals, the approver states
	that the message is the response to the query by the present database.
	"""
	approverId: Uuid!
	}
	"""
	Approval of data by third party with GnuPG signature that is created at some
	point in time. It becomes invalid when the approved data changes, even if the
	change is but a spelling-error correction in meta data like the description.
	In that case, the third party needs to re-create its approval. Unnecessary
	invalidation can be avoided by choosing the meta data that is part of the
	message being approved carefully.
	Steps to approve data:
	1. An institution adds data with meta information to a database.
	2. Some institution (may be the same) queries some meta information with
	a GraphQL query like
	```
	query {
	opticalData(id: <OPTICAL_DATA_ID>) {
	uuid
	name
	description
	resourceTree {
	root {
	value {
	hashValue
	locator
	dataFormatId
	}
	}
	}
	}
	}
	```
	and loads the data with an HTTPS GET request to the locator URL by running
	`wget <LOCATOR>`.
	3. The latter institution reviews whether some statement is true for the data,
	for example, whether the data conforms to some standard. If that is so, the
	institution queries the meta information to be signed which at least
	contains the hash value of the data
	```
	query {
	opticalData(id: <OPTICAL_DATA_ID>) {
	resourceTree {
	root {
	value {
	hashValue
	}
	}
	}
	}
	}
	```
	makes sure that the hash value is the same as before, wraps the statement
	and the meta information (named `<RESPONSE>` below) like so
	```
	{
	"statement": <STATEMENT>,
	"response": <RESPONSE>
	}
	```
	and signs this JSON data.
	4. The institution adds its approval of the data to some database or asks it to
	be added.
	Storing the GraphQL query, variables, and message with the signature is
	necessary because it needs to be known exactly which JSON data was signed (with
	exact whitespaces).
	Note that the query to be signed only includes paths that are part of the data
	and not part of some associates, which are actually all paths in the present
	GraphQL schema. However, the actual GraphQL schema of the database is in
	general an extension of the present GraphQL schema and may have paths to some
	associates like data format specifics of HTTP resources that it may fetch from
	the "metabase" and provides for convenience. These specifics should not be
	sigend as a change of, for example, the data format description would
	invalidate the signature although the data itself did not change.
	"""
	type DataApproval implements Approval {
	timestamp: DateTime!
	signature: String!
	keyFingerprint: String!
	query: GraphQlQuery!
	variables: Any!
	approverId: Uuid!
	"""
	Message that is signed. It is a string representation of the JSON
	```
	{
	"statement": <STATEMENT>,
	"response": <RESPONSE>
	}
	```
	with whitespaces as when the signature was created, where `<STATEMENT>` is
	the JSON of the field `statement` and `<RESPONSE>` is the JSON response of
	sending the GraphQL request given by the field `query`.
	"""
	message: String!
	"""
	Statement that is being made by the approver about the data identified by the
	query. The reference could for example be an ISO standard, which would mean
	that this approval states that the data conforms to this standard.
	"""
	statement: Reference!
	}
	"""
	Approval of response by database with
	[GnuPG](https://www.gnupg.org)
	signature.
	"""
	type ResponseApproval implements Approval {
	timestamp: DateTime!
	signature: String!
	keyFingerprint: String!
	query: GraphQlQuery!
	variables: Any!
	approverId: Uuid!
	"""
	Serialized response of the GraphQL query with whitespaces as when the
	signature was created.
	"""
	message: String!
	}

To support it minimally, the IGSDB would just need to have a one-to-many relationship from physical properties, optical properties, and detail files to one or three new tables that hold(s) the 8 scalar values of data approvals. To support it in a more complex fashion, it could validate approvals before they are added to the database and/or support the creation of such approvals for end users.