docs: metadata field and resource type process

zenodo · Apr 8, 2019 · f17321a · f17321a
1 parent 06fd61a
commit f17321a
Show file tree

Hide file tree

Showing 2 changed files with 51 additions and 56 deletions.
diff --git a/docs/development/process.rst b/docs/development/process.rst
@@ -209,68 +209,63 @@ use ``pip-tools`` to review and install all updated requirements.
     (zenodo-update)$ pip install pip-tools
     (zenodo-update)$ pip-compile
 
-Now manually update ``src/zenodo/requirements.txt`` with changes displayed
-in the diff.
-
-If an upgraded package causes issues, and the problem cannot easily be fixed,
-it should be moved from ``requirements.txt`` into
-``requirements.pinned.txt`` so it is clear which packages can easily be
-updated and which cannot.
-
 Expanding Zenodo metadata checklist
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Here you will find a short checklist/guide on how to add a new field to the metadata model, and what are the related
-files and models (ES mappings, UI deposit form), that need to be taken into consideration when such a change is made.
-
-1. Update the **Record** JSONSchema and ES mapping
-
-    a) Modify the Record JSONSchema files:
-
-        - Files JSONSchema - `records/jsonschemas/records/file_src-v1.0.0.json <https://github.com/zenodo/zenodo/blob/cce944e91e05720d0efbeb2bbe60cbf76d1a2286/zenodo/modules/records/jsonschemas/records/file_src-v1.0.0.json>`_
-        - Base JSONSchema (subschema of Record and Deposit) - `records/jsonschemas/records/base-v1.0.0.json <https://github.com/zenodo/zenodo/blob/cce944e91e05720d0efbeb2bbe60cbf76d1a2286/zenodo/modules/records/jsonschemas/records/base-v1.0.0.json>`_
-
-    b) **DO NOT** modify the following JSONSchemas by hand, as they need to be compiled from the sources (see point *a* above):
-
-        - Compile `records/jsonschemas/records/file-v1.0.0.json <https://github.com/zenodo/zenodo/blob/cce944e91e05720d0efbeb2bbe60cbf76d1a2286/zenodo/modules/records/jsonschemas/records/file-v1.0.0.json>`_ by running ``zenodo jsonschemas compilefile -d``.
-          **Optional - this step is necessary only if you modified the Files JSONSchema**.
-        - Compile `records/jsonschemas/records/record-v1.0.0.json <https://github.com/zenodo/zenodo/blob/cce944e91e05720d0efbeb2bbe60cbf76d1a2286/zenodo/modules/records/jsonschemas/records/record-v1.0.0.json>`_ by running ``zenodo jsonschemas compilerecord -d``
-        - Compile `deposit/jsonschemas/deposits/records/record-v1.0.0.json <https://github.com/zenodo/zenodo/blob/cce944e91e05720d0efbeb2bbe60cbf76d1a2286/zenodo/modules/deposit/jsonschemas/deposits/records/record-v1.0.0.json>`_ by running ``zenodo jsonschemas compiledeposit -d``
-
-    c) Update ES mappings
-
-        - Record ES mapping - `records/mappings/records/record-v1.0.0.json <https://github.com/zenodo/zenodo/blob/cce944e91e05720d0efbeb2bbe60cbf76d1a2286/zenodo/modules/records/mappings/records/record-v1.0.0.json>`_
-        - Deposit ES mapping - `deposit/mappings/deposits/records/record-v1.0.0.json <https://github.com/zenodo/zenodo/blob/cce944e91e05720d0efbeb2bbe60cbf76d1a2286/zenodo/modules/deposit/mappings/deposits/records/record-v1.0.0.json>`_
-
+Here you will find a short checklist/guide on how to add a new field to the
+metadata model, and what are the related files and models (ES mappings, UI
+deposit form), that need to be taken into consideration when such a change is
+made.
+
+1. Update the Record and Deposit JSONSchemas and ES mappings
+    a) To update any JSONSchema, you have to modify `records/jsonschemas/records/base-v1.0.0 <https://github.com/zenodo/zenodo/blob/master/zenodo/modules/records/jsonschemas/records/base-v1.0.0.json>`_ and then compile the rest of the JSONSchemas via the ``zenodo jsonschemas ...`` commands. **DO NOT** modify the following JSONSchemas by hand (unless you know what you're doing), as they need to be compiled from their sources (if you mess up there are tests in place that will catch any inconsistencies though)
+        - **Deposit**: Compile `deposit/jsonschemas/deposits/records/record-v1.0.0.json <https://github.com/zenodo/zenodo/blob/master/zenodo/modules/deposit/jsonschemas/deposits/records/record-v1.0.0.json>`_ by running ``zenodo jsonschemas compiledeposit -d``
+        - **Record**: Compile `records/jsonschemas/records/record-v1.0.0.json <https://github.com/zenodo/zenodo/blob/master/zenodo/modules/records/jsonschemas/records/record-v1.0.0.json>`_ by running ``zenodo jsonschemas compilerecord -d``
+    b) Update ES mappings by editing them directly
+        - **Record**: `records/mappings/records/record-v1.0.0.json <https://github.com/zenodo/zenodo/blob/master/zenodo/modules/records/mappings/records/record-v1.0.0.json>`_
+        - **Deposit**: `deposit/mappings/deposits/records/record-v1.0.0.json <https://github.com/zenodo/zenodo/blob/master/zenodo/modules/deposit/mappings/deposits/records/record-v1.0.0.json>`_
 2. Update Deposit and Record REST API (JSON serialisers/deserialisers)
-
-    - Common - `records/serializers/schemas/common.py <https://github.com/zenodo/zenodo/blob/cce944e91e05720d0efbeb2bbe60cbf76d1a2286/zenodo/modules/records/serializers/schemas/common.py>`_
-    - Deposit/Legacy - `records/serializers/schemas/legacyjson.py <https://github.com/zenodo/zenodo/blob/cce944e91e05720d0efbeb2bbe60cbf76d1a2286/zenodo/modules/records/serializers/schemas/legacyjson.py>`_
-    - New - `records/serializers/schemas/json.py <https://github.com/zenodo/zenodo/blob/cce944e91e05720d0efbeb2bbe60cbf76d1a2286/zenodo/modules/records/serializers/schemas/json.py>`_
-
+    - **Common**: `records/serializers/schemas/common.py <https://github.com/zenodo/zenodo/blob/master/zenodo/modules/records/serializers/schemas/common.py>`_
+    - **Deposit/Legacy**: `records/serializers/schemas/legacyjson.py <https://github.com/zenodo/zenodo/blob/master/zenodo/modules/records/serializers/schemas/legacyjson.py>`_
+    - **New**: `records/serializers/schemas/json.py <https://github.com/zenodo/zenodo/blob/master/zenodo/modules/records/serializers/schemas/json.py>`_
 3. Add to UI form (need to decide exactly where on how it should be displayed)
-
-    - Deposit Form JSONSchema - `deposit/static/json/zenodo_deposit/deposit_form.json <https://github.com/zenodo/zenodo/blob/cce944e91e05720d0efbeb2bbe60cbf76d1a2286/zenodo/modules/deposit/static/json/zenodo_deposit/deposit_form.json>`_
-    - Check if there are any Angular templates/directives: `deposit/static/templates/zenodo_deposit <https://github.com/zenodo/zenodo/tree/cce944e91e05720d0efbeb2bbe60cbf76d1a2286/zenodo/modules/deposit/static/templates/zenodo_deposit>`_ modifications required to implement the functionality of the new fields on the deposit form page
-
+    - **Deposit Form JSONSchema**: `deposit/static/json/zenodo_deposit/deposit_form.json <https://github.com/zenodo/zenodo/blob/master/zenodo/modules/deposit/static/json/zenodo_deposit/deposit_form.json>`_
+    - Check if there are any **Angular templates/directives** `deposit/static/templates/zenodo_deposit <https://github.com/zenodo/zenodo/tree/master/zenodo/modules/deposit/static/templates/zenodo_deposit>`_ modifications required to implement the functionality of the new fields on the deposit form page
 4. Serialization format updates
-
-    - DataCite - `records/serializers/schemas/datacite.py <https://github.com/zenodo/zenodo/blob/cce944e91e05720d0efbeb2bbe60cbf76d1a2286/zenodo/modules/records/serializers/schemas/datacite.py>`_
-
+    - **DataCite**: `records/serializers/schemas/datacite.py <https://github.com/zenodo/zenodo/blob/master/zenodo/modules/records/serializers/schemas/datacite.py>`_
         - `DataCite Metadata Schema v3.1 <https://schema.datacite.org/meta/kernel-3.1/>`_
-
-    - OpenAIRE JSON- `openaire/schema.py <https://github.com/zenodo/zenodo/blob/cce944e91e05720d0efbeb2bbe60cbf76d1a2286/zenodo/modules/openaire/schema.py>`_
-
-        - `OpenAIRE Schema <https://www.openaire.eu/schema/1.0/oaf-result-1.0.xsd>`_
-
-    - DublinCore - `records/serializers/schemas/dc.py <https://github.com/zenodo/zenodo/blob/cce944e91e05720d0efbeb2bbe60cbf76d1a2286/zenodo/modules/records/serializers/schemas/dc.py>`_
-
+        - `DataCite Metadata Schema v4.1 <https://schema.datacite.org/meta/kernel-4.1/>`_
+    - **DublinCore**: `records/serializers/schemas/dc.py <https://github.com/zenodo/zenodo/blob/master/zenodo/modules/records/serializers/schemas/dc.py>`_
         - `DCMI Metadata Terms <http://dublincore.org/documents/dcmi-terms/>`_
-
-    - CSL - `records/serializers/schemas/csl.py <https://github.com/zenodo/zenodo/blob/cce944e91e05720d0efbeb2bbe60cbf76d1a2286/zenodo/modules/records/serializers/schemas/csl.py>`_
+    - **OpenAIRE JSON**: `openaire/schema.py <https://github.com/zenodo/zenodo/blob/master/zenodo/modules/openaire/schema.py>`_
+        - `OpenAIRE Schema <https://www.openaire.eu/schema/1.0/oaf-result-1.0.xsd>`_
+    - **CSL**: `records/serializers/schemas/csl.py <https://github.com/zenodo/zenodo/blob/master/zenodo/modules/records/serializers/schemas/csl.py>`_
         - `CSL Terms <http://docs.citationstyles.org/en/stable/specification.html#appendix-ii-terms>`_
-    - BibTex - `records/serializers/bibtex.py <https://github.com/zenodo/zenodo/blob/cce944e91e05720d0efbeb2bbe60cbf76d1a2286/zenodo/modules/records/serializers/bibtex.py>`_
+    - **BibTex**: `records/serializers/bibtex.py <https://github.com/zenodo/zenodo/blob/master/zenodo/modules/records/serializers/bibtex.py>`_
         - `BibTeX documentation <http://ctan.math.washington.edu/tex-archive/biblio/bibtex/base/btxdoc.pdf>`_
-    - MARC21 - `records/serializers/schemas/marc21.py <https://github.com/zenodo/zenodo/blob/cce944e91e05720d0efbeb2bbe60cbf76d1a2286/zenodo/modules/records/serializers/schemas/marc21.py>`_
-
-5. Update `deposit REST API documentation <https://github.com/zenodo/developers.zenodo.org/blob/31497bdc1b0eb23f2a61c5858cddddc9a4955ae7/source/includes/resources/deposit/_representation.md>`_
+    - **MARC21**: `records/serializers/schemas/marc21.py <https://github.com/zenodo/zenodo/blob/master/zenodo/modules/records/serializers/schemas/marc21.py>`_
+    - **JSON-LD (schema.org)**: `records/serializers/schemas/schemaorg.py <https://github.com/zenodo/zenodo/blob/master/zenodo/modules/records/serializers/schemas/schemaorg.py>`_
+5. Update `deposit REST API documentation <https://github.com/zenodo/developers.zenodo.org/blob/master/source/includes/resources/deposit/_representation.md>`_
+
+Adding Resource Types
+^^^^^^^^^^^^^^^^^^^^^
+
+Resource types follow a two-level hierarchy of ``<type> -> <sub-type>`` (e.g.
+"Image" -> "Figure"). Especially when adding (or updating) resource types (e.g.
+a new "Publication" sub-type), besides checking the above places, you have to
+also update the `records/data/objecttypes.json
+<https://github.com/zenodo/zenodo/blob/master/zenodo/modules/records/data/objecttypes.json>`_
+file accordingly with a new entry:
+
+- **internal_id**: Unique internal ID for the type. Basically ``<type>`` (e.g. ``software``) or ``<type>-<sub-type>`` (e.g. ``image-figure``).
+- **id**: JSONSchema ID field. Basically ``https://zenodo.org/objecttypes/<type>`` or ``https://zenodo.org/objecttypes/<type>/<sub-type>``. Example ``https://zenodo.org/objecttypes/publication/softwaredocumentation#``
+- **parent**: If a ``sub-type``, this points to the ``type``'s ``id``. Example: ``{"$ref": "https://zenodo.org/objecttypes/publication"}``
+- **title**: Title for the type. Example: ``{"en": "Software documentation"}``
+- **title_plural**: Title in plural. Example: ``{"en": "Software documentation"}``
+- **schema.org**: Schema.org DataType, should be CreativeWork or one of its appropriate subtypes (`vocabulary <https://schema.org/CreativeWork#subtypes>`__). Example: ``https://schema.org/CreativeWork``
+- **datacite**: DataCite ResourceType (`vocabulary <https://schema.datacite.org/meta/kernel-4.1/>`__). Example: ``{"general": "Text", "type": "Software documentation"}``
+- **eurepo**: ``info-eu-repo`` type (`vocabulary <https://wiki.surfnet.nl/display/standards/info-eu-repo#info-eu-repo-Publicationtypes>`__). Example: ``info:eu-repo/semantics/technicalDocumentation``
+- **openaire**: OpenAIRE-specific fields
+    - **resourceType**: OpenAIRE resource type code (`vocabulary <https://issue.openaire.research-infrastructures.eu/issues/2938>`__). Example: ``0009``
+    - **type**: OpenAIRE type used for direct indexing. Can be ``publication``, ``dataset``, ``software`` or ``other``
+- **csl**: Citation Style Language type (`vocabulary <http://docs.citationstyles.org/en/stable/specification.html#appendix-iii-types>`__). Example: ``article``
diff --git a/docs/development/projectlifecycle.rst b/docs/development/projectlifecycle.rst
@@ -48,7 +48,7 @@ The beginning of each sprint cycle roughly follows the following pattern:
     - Move open milestone issues to backlog (``1 - Todo`` -> ``0 - Backlog``).
     - Close milestone (progress should be 100%).
 * Requirements update
-    - Update Python package requirements. Needed in order to ensure security fixes from packages are integrated (see :ref:`updateing_requirements`).
+    - Update Python package requirements. Needed in order to ensure security fixes from packages are integrated (see :ref:`updating_requirements`).
 * Sprint planning
     - Create new milestone and assign end-date.
     - Assign issues to milestone (see below).