Include Changes for specification V2.0.0 #105

HLWeil · 2024-06-06T08:28:42Z

This PR merges all changes accumulated for the Version 2.0.0 of the specification into the main branch. Afterwards, Version 2.0.0 will be released.

Big Overhaul of data representation by @HLWeil
Mechanisms for ARC Quality Control section by @kMutagene
Various changes to xlsx table representation
Make study section in investigaiton file optional
- Make some sections in investigation and study metadata sheets optional #101
Various clarifications and fixes

Input would be very welcome, @kappe-c @floWetzels @ZimmerD @kMutagene @Freymaurer @chgarth @muehlhaus @gdoniparthi

fix typo in STUDY CONTACTS table section

Make arc.cwl optional

Add data path annotation section to ARC specification

Fix typos in isa-xlsx

Add comment to annotation table

Validation specs

Rework Data Nodes

Add versioning of validation packages to validation_packages.yml

…der structure

Updated ARC RO-Crate Profile Description

let git do the cqc versioning, add optional version suffix in cqc folder structure

Make some sections in investigation and study metadata sheets optional

Datamap specification

add `arc_specification` key to `validation_packages.yml`

Freymaurer · 2024-06-06T09:41:13Z

ARC specification.md


 Each ARC is a directory containing the following elements:

 - *Studies* are collections of material and resources used within the investigation.
-Metadata that describe the characteristics of material and resources follow the ISA study model. Study-level metadata is stored in [ISA-XLSX](#isa-xlsx-format) format in a file `isa.study.xlsx`, which MUST exist to specify the input material or data resources. Resources MAY include biological materials (e.g. plant samples, analytical standards) created during the current investigation. Resources MAY further include external data (e.g., knowledge files, results files) that need to be included and cannot be referenced due to external limitations. Resources described in a study file can be the input for one or multiple assays. Further details on `isa.study.xlsx` are specified [below](#study-and-resources). Resource (descriptor) files MUST be placed in a `resources` subdirectory.
+Metadata that describe the characteristics of material and resources follow the ISA study model. Study-level metadata is stored in [ISA-XLSX](#isa-xlsx-format) format in a `isa.study.xlsx` file, which MUST exist to specify the input material or data resources. Resources MAY include biological materials (e.g. plant samples, analytical standards) created during the current investigation. Resources MAY further include external data (e.g., knowledge files, results files) that need to be included and cannot be referenced due to external limitations. Resources described in a study file can be the input for one or multiple assays. Further details on `isa.study.xlsx` are specified [below](#study-and-resources). Resource (descriptor) files MUST be placed in a `resources` subdirectory. Further explications about data entities defined in the study MAY be stored in [ISA-XLSX](#isa-xlsx-format) format in a `isa.datamap.xlsx` file, which SHOULD exist for studies containing data. Further details on `isa.datamap.xlsx` are specified [in the isa-xlsx specification](ISA-XLSX.md#datamap-file).


Metadata that describe the characteristics of material and resources follow the ISA study model.

For me this sentence reads strangely

Yeah I think it was replaced but not cut out. Will throw it out now

Freymaurer · 2024-06-06T09:41:45Z

ARC specification.md


- *Assays* correspond to outcomes of experimental assays or analytical measurements (in the interpretation of the ISA model) and are treated as immutable data. Each assay is a collection of files, together with a corresponding metadata file, stored in a subdirectory of the top-level subdirectory `assays`. Assay-level metadata is stored in [ISA-XLSX](#isa-xlsx-format) format in a file `isa.assay.xlsx`, which MUST exist for each assay. Further details on `isa.assay.xlsx` are specified [below](#assay-data-and-metadata). Assay data files MUST be placed in a `dataset` subdirectory.
+- *Assays* correspond to outcomes of experimental assays or analytical measurements (in the interpretation of the ISA model) and are treated as immutable data. Each assay is a collection of files, together with a corresponding metadata file, stored in a subdirectory of the top-level subdirectory `assays`. Assay-level metadata is stored in [ISA-XLSX](#isa-xlsx-format) format in a `isa.assay.xlsx` file, which MUST exist for each assay. Further details on `isa.assay.xlsx` are specified [below](#assay-data-and-metadata). Assay data files MUST be placed in a `dataset` subdirectory. Further explications about data entities defined in the assay MAY be stored in [ISA-XLSX](#isa-xlsx-format) format in a `isa.datamap.xlsx` file, which SHOULD exist for each assay. Further details on `isa.datamap.xlsx` are specified [in the isa-xlsx specification](ISA-XLSX.md#datamap-file).


Further explications about data entities defined in the assay MAY be stored in ISA-XLSX format in a isa.datamap.xlsx file, which SHOULD exist for each assay. Further details on isa.datamap.xlsx are specified in the isa-xlsx specification

Mixing MAY and SHOULD? maybe unify?

I think the way it is currently written holds two questions (and imho in the wrong order).

Should such metadata be stored?

If so, where should it be stored?

Suggestion changing the order and using only one keyword (SHOULD):

Further explications about data entities defined in the assay SHOULD exist for each assay, in ISA-XLSX format in a isa.datamap.xlsx file. Further details ...

It was meant the following way:

In general, each assay or study SHOULD contain a datamap file

For every data entity you MAY decide to add additional information to this datamap file

But I agree it reads like there are options for how to store additional information.

Freymaurer · 2024-06-06T09:42:16Z

ARC specification.md

            \--- resources 
            \--- protocol [optional / add. payload]
 \--- assays
    \--- <assay_name> 
            |    isa.assay.xlsx  
+            |    isa.datamap.xlsx  


[optional]?

Freymaurer · 2024-06-06T09:59:04Z

ARC specification.md

@@ -142,12 +156,16 @@ The `study` file MUST follow the [ISA-XLSX study file specification](ISA-XLSX.md

 Protocols that are necessary to describe the sample or material creating process can be placed under the protocols directory.

-### Assay Data and Metadata
+Further explications about data entities defined in the assay MAY be stored in [ISA-XLSX](#isa-xlsx-format) format in a `isa.datamap.xlsx` file, which SHOULD exist for each assay. Further details on `isa.datamap.xlsx` are specified [in the isa-xlsx specification](ISA-XLSX.md#datamap-file).


same as above (switch beween MAY and SHOULD)

Freymaurer · 2024-06-06T09:59:14Z

ARC specification.md


 All measurement data sets are considered as assays and are considered immutable input data. Assay data MUST be placed into a unique subdirectory of the top-level `assays` subdirectory. All ISA metadata specific to a single assay MUST be annotated in the file `isa.assay.xlsx` at the root of the assay's subdirectory. This workbook MUST contain a single assay that can be organized in one or multiple worksheets. 

 The `assay` file MUST follow the [ISA-XLSX assay file specification](ISA-XLSX.md#assay-file).

+Further explications about data entities defined in the assay MAY be stored in [ISA-XLSX](#isa-xlsx-format) format in a `isa.datamap.xlsx` file, which SHOULD exist for each assay. Further details on `isa.datamap.xlsx` are specified [in the isa-xlsx specification](ISA-XLSX.md#datamap-file).


same as above (switch beween MAY and SHOULD)

Freymaurer · 2024-06-06T10:01:15Z

ARC specification.md

+
+`assays/Assay2/isa.assay.xlsx`:
+
+| Input [Data] | Parameter[script file]          | Output [Data] |


Component [script file]? 😅

Freymaurer · 2024-06-06T10:01:57Z

ARC specification.md

+  - Data nodes in `isa.assay.xlsx` files: The path MAY be specified relative to the `dataset` sub-folder of the assay
+  - Data nodes in `isa.study.xlsx` files: The path MAY be specified relative to the `resources` sub-folder of the study
+
+### Examples


Can we also get example for folder specific pattern?

Freymaurer · 2024-06-06T10:29:02Z

ISA-XLSX.md

+| Comment [Answer to everything] |
+|--------------------------------|
+|                      forty-two |
+
 ## Others

 Columns whose headers do not follow any of the formats described above are considered additional payload and are out of the scope of this specification.


Does this mean we now officially support free text columns?

I'd read that as "the tool implementing the standard is free to decide what to do with free text columns"

Yes, just a heads-up that a hard fail on unknown columns is not necessary.

Freymaurer · 2024-06-06T10:29:51Z

ISA-XLSX.md

+In the `Datamap Table sheets`, column headers MUST have the first letter of each word in upper case, with the exception of the referencing label (REF).
+
+The content of the datamap table MUST be placed in an `xlsx table` whose name starts with `datamapTable`. Each sheet MUST contain at most one such annotation table. Only cells inside this table are considered as part of the formatted metadata.


placed in an xlsx table whose name starts with

start with or equal?

Hmm, wanted to keep it in line with the AnnotationTable. But I agree it's not necessary as there can only be one table. Will change to equals.

Freymaurer · 2024-06-06T10:32:01Z

ISA-XLSX.md

+## Comments
+
+A `Comment` can be used to provide some additional information. Columns headed with `Comment[<comment name>]` MAY appear anywhere in the Annotation Table. The comment always refers to the Annotation Table. The value MUST be free text.


So we must remember Comment location in ARCtrl?

Not necessarily, as the comment column does not refere to a specific other column but always to the table as a whole.

Add validation_summary.json specs, package metadata, and ARC apps

HLWeil · 2024-06-10T13:28:41Z

@Freymaurer @kappe-c @kMutagene
Thanks a lot for your input!

I made some changes according to your comments. Please check again if your remarks are resolved now!

HLWeil and others added 30 commits November 17, 2023 12:03

xlsx: small adjustments to inputs and outputs

5acfed0

loosen up arc.cwl constraint from MUST to SHOULD

aeae73e

add comment to annotation table

7d5df0e

add data path annotation section to ARC specification

a9bd993

small change to data path annotation specification

0d83c77

clarify constraint for folder specific pattern

8c2bd48

fix a typo in the STUDY CONTACTS table section

0acd515

Merge pull request #90 from nfdi4plants/patch-2

94bb4f7

fix typo in STUDY CONTACTS table section

Merge pull request #83 from nfdi4plants/arc_cwl

8f8b287

Make arc.cwl optional

Merge pull request #85 from nfdi4plants/data_paths

3e5ef5d

Add data path annotation section to ARC specification

fix typos in isa-xlsx

ae969c2

Merge pull request #91 from nfdi4plants/typos

84aa6d6

Fix typos in isa-xlsx

make annotation table comment refer only to table

591c24c

Merge pull request #84 from nfdi4plants/comment

574f5dc

Add comment to annotation table

start inclusion of data selectors with some links

b336710

finish up first draft of new data columns

91644ed

adjust examples according to data specification changes

6a4eefa

change Data Selector to Data Selector Format

1f6e246

wip validation specs

f02c087

add validation branch and targets file specs

0831e8d

typo fix

5f3acff

add missing arc-validate link

895261f

add a reference to git manual regarding orphan branches

d2a7d68

remove unnecessary space

c385cae

Add Validation section, Fix header levels

350f07c

spelling fixes

9fa4574

fix missing link

115e8c7

Merge pull request #95 from nfdi4plants/validation-specs

01d5483

Validation specs

Merge pull request #93 from nfdi4plants/selector

604a083

Rework Data Nodes

Add versioning of validation packages to validation_packages.yml

280fcf2

kMutagene and others added 16 commits February 16, 2024 14:32

Merge pull request #96 from nfdi4plants/validation-specs

57cbbcf

Add versioning of validation packages to validation_packages.yml

let git do the cqc versioning, add optional version suffix in cqc fol…

077475a

…der structure

Updated ARC RO-Crate Profile Description

b66322d

fixed typos in RO-Crate profile description

dd7aa52

Merge pull request #100 from floWetzels/v2.0.0

286053e

Updated ARC RO-Crate Profile Description

Merge pull request #98 from nfdi4plants/validation-specs

17ac78a

let git do the cqc versioning, add optional version suffix in cqc folder structure

make some sections in investigation and study metadata sheets optional

4126940

Merge pull request #101 from nfdi4plants/isa-light

4aa6718

Make some sections in investigation and study metadata sheets optional

add arc_specification key to validation_packages.yml

afc66e9

add Datamap Table sheets section

a5a029c

add datamap file section to isa-xlsx

e8dd204

reference datamap file from arc assay and study specifications

41d513f

add manage-issues github workflow

3329352

Merge pull request #104 from nfdi4plants/datamap

2eb7da6

Datamap specification

Merge pull request #102 from nfdi4plants/validation-specs

070a9c2

add `arc_specification` key to `validation_packages.yml`

Add validation_summary.json specs, package metadata, and ARC apps

4b4c29c

Freymaurer reviewed Jun 6, 2024

View reviewed changes

kMutagene and others added 3 commits June 7, 2024 16:15

typo fix

53439ef

Merge pull request #108 from nfdi4plants/validation-specs

ce7caab

Add validation_summary.json specs, package metadata, and ARC apps

small changes and additions according to PR review for v2.0.0

811870e

HLWeil merged commit 4ce4fb8 into main Jun 11, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Include Changes for specification V2.0.0 #105

Include Changes for specification V2.0.0 #105

HLWeil commented Jun 6, 2024

Freymaurer Jun 6, 2024

HLWeil Jun 10, 2024

Freymaurer Jun 6, 2024

kappe-c Jun 7, 2024

HLWeil Jun 10, 2024

HLWeil Jun 10, 2024 •

edited

Loading

Freymaurer Jun 6, 2024

Freymaurer Jun 6, 2024

Freymaurer Jun 6, 2024

Freymaurer Jun 6, 2024

Freymaurer Jun 6, 2024

Freymaurer Jun 6, 2024

kMutagene Jun 10, 2024

HLWeil Jun 10, 2024

Freymaurer Jun 6, 2024

HLWeil Jun 10, 2024

Freymaurer Jun 6, 2024

HLWeil Jun 10, 2024

HLWeil commented Jun 10, 2024


		- Assays correspond to outcomes of experimental assays or analytical measurements (in the interpretation of the ISA model) and are treated as immutable data. Each assay is a collection of files, together with a corresponding metadata file, stored in a subdirectory of the top-level subdirectory `assays`. Assay-level metadata is stored in [ISA-XLSX](#isa-xlsx-format) format in a file `isa.assay.xlsx`, which MUST exist for each assay. Further details on `isa.assay.xlsx` are specified [below](#assay-data-and-metadata). Assay data files MUST be placed in a `dataset` subdirectory.
		- Assays correspond to outcomes of experimental assays or analytical measurements (in the interpretation of the ISA model) and are treated as immutable data. Each assay is a collection of files, together with a corresponding metadata file, stored in a subdirectory of the top-level subdirectory `assays`. Assay-level metadata is stored in [ISA-XLSX](#isa-xlsx-format) format in a `isa.assay.xlsx` file, which MUST exist for each assay. Further details on `isa.assay.xlsx` are specified [below](#assay-data-and-metadata). Assay data files MUST be placed in a `dataset` subdirectory. Further explications about data entities defined in the assay MAY be stored in [ISA-XLSX](#isa-xlsx-format) format in a `isa.datamap.xlsx` file, which SHOULD exist for each assay. Further details on `isa.datamap.xlsx` are specified [in the isa-xlsx specification](ISA-XLSX.md#datamap-file).


		`assays/Assay2/isa.assay.xlsx`:

		\| Input [Data] \| Parameter[script file] \| Output [Data] \|

		In the `Datamap Table sheets`, column headers MUST have the first letter of each word in upper case, with the exception of the referencing label (REF).

		The content of the datamap table MUST be placed in an `xlsx table` whose name starts with `datamapTable`. Each sheet MUST contain at most one such annotation table. Only cells inside this table are considered as part of the formatted metadata.

		## Comments

		A `Comment` can be used to provide some additional information. Columns headed with `Comment[<comment name>]` MAY appear anywhere in the Annotation Table. The comment always refers to the Annotation Table. The value MUST be free text.

Include Changes for specification V2.0.0 #105

Include Changes for specification V2.0.0 #105

Conversation

HLWeil commented Jun 6, 2024

This PR merges all changes accumulated for the Version 2.0.0 of the specification into the main branch. Afterwards, Version 2.0.0 will be released.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HLWeil Jun 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HLWeil commented Jun 10, 2024

HLWeil Jun 10, 2024 •

edited

Loading