ValTaxonomicAPIWoRMS_ValTaxonomicaAPIWoRMS.txt

--------------------------------------------------------------------------------
---Do not copy /No Copiar

--ENGLISH VERSION

SiB Colombia scripts for Biodiversity data validation and cleaning in OPEN REFINE
https://github.com/SIB-Colombia/data-quality-open-refine

Script Taxonomic Validation using Worms's API
Language: GREL 'General Refine Expression Language'
Created: 2019-05-01
Last Update:2019-06-12
Contributors: Camila Plata, Leonardo Buitrago

This script:
1-Matches original scientificName with Worms's taxonomic Backbone
2-Retrieves Worms's rank and status allowing the user to evaluate the state of each name
3-Retrieves Worms's higher taxonomy for all names
4-Retrieves Worms's habitat classification in DwC
5-Compares Worms's taxonomic suggestions with original taxonomy using a boolean descriptor (1,0)

			Conventions boolean descriptor
			0-Worms's suggested name DOES NOT match the original name
			1-Worms's suggested name matches the original name

Conditions
Dataset with minimum the DwC element 'scientificName'
To obtain a validation of higher taxonomy, these DwC elements are also required: 'kingdom','phylum','class','order','family','genus'

New data will be stored in columns at the beginning of the dataset
Taxonomy elements are reorganized to facilitate the taxonomic validation

--VERSION EN ESPAÑOL

Rutinas del SiB Colombia para la validación y limpieza de datos primarios 
de Biodiversidad en OPEN REFINE
https://github.com/SIB-Colombia/data-quality-open-refine

Nombre rutina: Validación taxonomica usando el API de Worms
Languaje: GREL 'General Refine Expression Language'
Creado:2019-05-01
Última Actualización:2019-06-12
Autores: Camila Plata, Leonardo Buitrago

Esta rutina:
1-Compara el elemento scientificName con el árbol taxonómico de Worms
2-Trae los elementos rank y status permitiendo al usuario evaluar el estado del nombre científico
3-Trae la taxonomia superior de cada nombre de acuerdo a Worms
4-Trae la clasificación del habitat de cada nombre en DwC según Worms
5-Compara las sugerencias taxonómicas de Worms con la taxonomia documentada usando descriptores booleanos (1,0)

			Conventions boolean descriptor
			0- El nombre sugerido por Worms NO coincide con el nombre documentado en el conjunto de datos
			1- El nombre sugerido por Worms coincide con el nombre documentado en el conjunto de datose

Requerimientos:
Conjunto de datos con minimo el elemento DwC 'scientificName' documentado
Para obtener la validación de la taxonomía superior también se requieren los elementos DwC: 'kingdom','phylum','class','order','family','genus'

Los nuevos datos seran guardados en columnas el inicio del conjunto de datos
Los elementos taxonómicos son reorganizados para facilitar la validación taxonómicas

---Do not copy /No Copiar
--------------------------------------------------------------------------------

[
  {
    "op": "core/text-transform",
    "description": "Text transform on cells in column scientificName",
    "engineConfig": {
      "mode": "row-based",
      "facets": []
    },
    "columnName": "scientificName",
    "expression": "grel:value.replace(/\\s+/,' ').trim()",
    "onError": "keep-original",
    "repeat": false,
    "repeatCount": 10
  },
  {
    "op": "core/column-addition",
    "engineConfig": {
      "facets": [],
      "mode": "row-based"
    },
    "baseColumnName": "scientificName",
    "expression": "grel:value.replace(' ','%20')",
    "onError": "set-to-blank",
    "newColumnName": "NomAPIw",
    "columnInsertIndex": 2,
    "description": "Create column NomAPIw at index 4 based on column scientificName using expression grel:value.replace(' ','%20')"
  },
 {
    "op": "core/column-addition-by-fetching-urls",
    "description": "Create column c at index 3 by fetching URLs based on column NomAPIw",
    "engineConfig": {
      "facets": [],
      "mode": "row-based"
    },
    "newColumnName": "callAPIworms",
    "columnInsertIndex": 3,
    "baseColumnName": "NomAPIw",
    "urlExpression": "grel:'http://www.marinespecies.org/rest/AphiaRecordsByMatchNames?scientificnames%5B%5D='+value+'&marine_only=false'",
    "onError": "set-to-blank",
    "delay": 0,
    "cacheResponses": true,
    "httpHeadersJson": [
      {
        "name": "authorization",
        "value": ""
      },
      {
        "name": "user-agent",
        "value": "OpenRefine 3.1 [b90e413]"
      },
      {
        "name": "accept",
        "value": "*/*"
      }
    ]
  },
  {
    "op": "core/text-transform",
    "description": "Text transform on cells in column callAPIworms",
    "engineConfig": {
      "facets": [],
      "mode": "row-based"
    },
    "columnName": "callAPIworms",
    "expression": "grel:value.replace('[','').replace(']','')",
    "onError": "keep-original",
    "repeat": false,
    "repeatCount": 10
  },
  {
    "op": "core/column-addition",
    "description": "Create column isTerrestrial at index 2 based on column callAPIworms",
    "engineConfig": {
      "facets": [],
      "mode": "row-based"
    },
    "newColumnName": "isTerrestrial",
    "columnInsertIndex": 2,
    "baseColumnName": "callAPIworms",
    "expression": "grel:value.parseJson().get('isTerrestrial').toString()",
    "onError": "set-to-blank"
  },
  {
    "op": "core/column-addition",
    "description": "Create column isFreshwater at index 2 based on column callAPIworms",
    "engineConfig": {
      "facets": [],
      "mode": "row-based"
    },
    "newColumnName": "isFreshwater",
    "columnInsertIndex": 2,
    "baseColumnName": "callAPIworms",
    "expression": "grel:value.parseJson().get('isFreshwater').toString()",
    "onError": "set-to-blank"
  },
  {
    "op": "core/column-addition",
    "description": "Create column isBrackish at index 2 based on column callAPIworms",
    "engineConfig": {
      "facets": [],
      "mode": "row-based"
    },
    "newColumnName": "isBrackish",
    "columnInsertIndex": 2,
    "baseColumnName": "callAPIworms",
    "expression": "grel:value.parseJson().get('isBrackish').toString()",
    "onError": "set-to-blank"
  },
  {
    "op": "core/column-addition",
    "description": "Create column isMarine at index 2 based on column callAPIworms",
    "engineConfig": {
      "facets": [],
      "mode": "row-based"
    },
    "newColumnName": "isMarine",
    "columnInsertIndex": 2,
    "baseColumnName": "callAPIworms",
    "expression": "grel:value.parseJson().get('isMarine').toString()",
    "onError": "set-to-blank"
  }, 
  {
    "op": "core/column-addition",
    "description": "Create column A at index 2 based on column callAPIworms",
    "engineConfig": {
      "facets": [],
      "mode": "row-based"
    },
    "newColumnName": "nameAccordingToID",
    "columnInsertIndex": 2,
    "baseColumnName": "callAPIworms",
    "expression": "grel:value.parseJson().get('url')",
    "onError": "set-to-blank"
  },  
  {
    "op": "core/column-addition",
    "description": "Create column A at index 2 based on column callAPIworms",
    "engineConfig": {
      "facets": [],
      "mode": "row-based"
    },
    "newColumnName": "nameAccordingTo",
    "columnInsertIndex": 2,
    "baseColumnName": "callAPIworms",
    "expression": "grel:value.parseJson().get('citation')",
    "onError": "set-to-blank"
  },
   {
    "op": "core/column-move",
    "description": "Move column infraspecificEpithet to position 2",
    "columnName": "infraspecificEpithet",
    "index": 2
  },
  {
    "op": "core/column-addition",
    "description": "Create column specificEpithetSuggested at index 2 based on column callAPIworms",
    "engineConfig": {
      "mode": "row-based",
      "facets": []
    },
    "newColumnName": "specificEpithetSuggested",
    "columnInsertIndex": 2,
    "baseColumnName": "callAPIworms",
    "expression": "grel:split(value.parseJson().get(\"scientificname\"),' ')[1]",
    "onError": "set-to-blank"
  },
  {
    "op": "core/column-addition",
    "description": "Create column specificEpithetValidation at index 2 based on column suggestedSpecificEpithet",
    "engineConfig": {
      "mode": "row-based",
      "facets": []
    },
    "newColumnName": "specificEpithetValidation",
    "columnInsertIndex": 2,
    "baseColumnName": "specificEpithetSuggested",
    "expression": "grel:if(value==cells[\"specificEpithet\"].value,'1','0').toString()",
    "onError": "set-to-blank"
  },
  {
    "op": "core/column-move",
    "description": "Move column infraspecificEpithet to position 2",
    "columnName": "specificEpithet",
    "index": 2
  },
  {
    "op": "core/column-addition",
    "description": "Create column A at index 2 based on column callAPIworms",
    "engineConfig": {
      "facets": [],
      "mode": "row-based"
    },
    "newColumnName": "genusSuggested",
    "columnInsertIndex": 2,
    "baseColumnName": "callAPIworms",
    "expression": "grel:value.parseJson().get('genus')",
    "onError": "set-to-blank"
  },
  {
    "op": "core/column-addition",
    "description": "Create column genusValidation at index 2 based on column suggstedKingdom",
    "engineConfig": {
      "mode": "row-based",
      "facets": []
    },
    "newColumnName": "genusValidation",
    "columnInsertIndex": 2,
    "baseColumnName": "genusSuggested",
    "expression": "grel:if(value==cells[\"genus\"].value,'1','0').toString()",
    "onError": "set-to-blank"
  },
   {
    "op": "core/column-move",
    "description": "Move column genus to position 2",
    "columnName": "genus",
    "index": 2
  },
  {
    "op": "core/column-addition",
    "description": "Create column familySuggested at index 2 based on column callAPIworms",
    "engineConfig": {
      "facets": [],
      "mode": "row-based"
    },
    "newColumnName": "familySuggested",
    "columnInsertIndex": 2,
    "baseColumnName": "callAPIworms",
    "expression": "grel:value.parseJson().get('family')",
    "onError": "set-to-blank"
  },
  {
    "op": "core/column-addition",
    "description": "Create column familyValidation at index 2 based on column familySuggested",
    "engineConfig": {
      "mode": "row-based",
      "facets": []
    },
    "newColumnName": "familyValidation",
    "columnInsertIndex": 2,
    "baseColumnName": "familySuggested",
    "expression": "grel:if(value==cells[\"family\"].value,'1','0').toString()",
    "onError": "set-to-blank"
  },
   {
    "op": "core/column-move",
    "description": "Move column family to position 2",
    "columnName": "family",
    "index": 2
  },
     {
    "op": "core/column-addition",
    "description": "Create column A at index 2 based on column callAPIworms",
    "engineConfig": {
      "facets": [],
      "mode": "row-based"
    },
    "newColumnName": "orderSuggested",
    "columnInsertIndex": 2,
    "baseColumnName": "callAPIworms",
    "expression": "grel:value.parseJson().get('order')",
    "onError": "set-to-blank"
  },
  {
    "op": "core/column-addition",
    "description": "Create column orderValidation at index 2 based on column suggestedOrder",
    "engineConfig": {
      "mode": "row-based",
      "facets": []
    },
    "newColumnName": "orderValidation",
    "columnInsertIndex": 2,
    "baseColumnName": "orderSuggested",
    "expression": "grel:if(value==cells[\"order\"].value,'1','0').toString()",
    "onError": "set-to-blank"
  },
   {
    "op": "core/column-move",
    "description": "Move column order to position 2",
    "columnName": "order",
    "index": 2
  },
   {
    "op": "core/column-addition",
    "description": "Create column A at index 2 based on column callAPIworms",
    "engineConfig": {
      "facets": [],
      "mode": "row-based"
    },
    "newColumnName": "classSuggested",
    "columnInsertIndex": 2,
    "baseColumnName": "callAPIworms",
    "expression": "grel:value.parseJson().get('class')",
    "onError": "set-to-blank"
  },
  {
    "op": "core/column-addition",
    "description": "Create column classValidation at index 2 based on column suggestedClass",
    "engineConfig": {
      "mode": "row-based",
      "facets": []
    },
    "newColumnName": "classValidation",
    "columnInsertIndex": 2,
    "baseColumnName": "classSuggested",
    "expression": "grel:if(value==cells[\"class\"].value,'1','0').toString()",
    "onError": "set-to-blank"
  },
   {
    "op": "core/column-move",
    "description": "Move column class to position 2",
    "columnName": "class",
    "index": 2
  },
  {
    "op": "core/column-addition",
    "description": "Create column A at index 2 based on column callAPIworms",
    "engineConfig": {
      "facets": [],
      "mode": "row-based"
    },
    "newColumnName": "phylumSuggested",
    "columnInsertIndex": 2,
    "baseColumnName": "callAPIworms",
    "expression": "grel:value.parseJson().get('phylum')",
    "onError": "set-to-blank"
  },   
   {
    "op": "core/column-addition",
    "description": "Create column phylumValidation at index 2 based on column suggestedPhylum",
    "engineConfig": {
      "mode": "row-based",
      "facets": []
    },
    "newColumnName": "phylumValidation",
    "columnInsertIndex": 2,
    "baseColumnName": "phylumSuggested",
    "expression": "grel:if(value==cells[\"phylum\"].value,'1','0').toString()",
    "onError": "set-to-blank"
  },
   {
    "op": "core/column-move",
    "description": "Move column phylum to position 2",
    "columnName": "phylum",
    "index": 2
  },
  {
    "op": "core/column-addition",
    "description": "Create column A at index 2 based on column callAPIworms",
    "engineConfig": {
      "facets": [],
      "mode": "row-based"
    },
    "newColumnName": "kingdomSuggested",
    "columnInsertIndex": 2,
    "baseColumnName": "callAPIworms",
    "expression": "grel:value.parseJson().get('kingdom')",
    "onError": "set-to-blank"
  },
   {
    "op": "core/column-addition",
    "description": "Create column kingdomValidation at index 2 based on column suggestedKingdom",
    "engineConfig": {
      "mode": "row-based",
      "facets": []
    },
    "newColumnName": "kingdomValidation",
    "columnInsertIndex": 2,
    "baseColumnName": "kingdomSuggested",
    "expression": "grel:if(value==cells[\"kingdom\"].value,'1','0').toString()",
    "onError": "set-to-blank"
  },
   {
    "op": "core/column-move",
    "description": "Move column kingdom to position 2",
    "columnName": "kingdom",
    "index": 2
  },
  {
    "op": "core/column-addition",
    "description": "Create column suggestedTaxonomicStatus at index 2 based on column callAPIworms",
    "engineConfig": {
      "mode": "row-based",
      "facets": []
    },
    "newColumnName": "taxonomicStatusSuggested",
    "columnInsertIndex": 2,
    "baseColumnName": "callAPIworms",
    "expression": "grel:value.parseJson().get('status')",
    "onError": "set-to-blank"
  }, 
  {
    "op": "core/column-addition",
    "description": "Create column A at index 2 based on column callAPIworms",
    "engineConfig": {
      "facets": [],
      "mode": "row-based"
    },
    "newColumnName": "taxonRankSuggested",
    "columnInsertIndex": 2,
    "baseColumnName": "callAPIworms",
    "expression": "grel:value.parseJson().get('rank')",
    "onError": "set-to-blank"
  },
  {
    "op": "core/column-addition",
    "description": "Create column A at index 2 based on column callAPIworms",
    "engineConfig": {
      "facets": [],
      "mode": "row-based"
    },
    "newColumnName": "scientificNameAuthorshipSuggested",
    "columnInsertIndex": 2,
    "baseColumnName": "callAPIworms",
    "expression": "grel:value.parseJson().get('authority')",
    "onError": "set-to-blank"
  },
  {
    "op": "core/column-addition",
    "description": "Create column authorshipValidation at index 2 based on column suggstedAuthorship",
    "engineConfig": {
      "mode": "row-based",
      "facets": []
    },
    "newColumnName": "scientificNameAuthorshipValidation",
    "columnInsertIndex":2,
    "baseColumnName": "scientificNameAuthorshipSuggested",
    "expression": "grel:if(value==cells[\"scientificNameAuthorship\"].value,'1','0').toString()",
    "onError": "set-to-blank"
  },
  {
    "op": "core/column-move",
    "description": "Move column scientificNameAuthorship to position 2",
    "columnName": "scientificNameAuthorship",
    "index": 2
  },
  {
    "op": "core/column-addition",
    "description": "Create column acceptedScientificName(species) at index 2 based on column callAPIworms",
    "engineConfig": {
      "mode": "row-based",
      "facets": []
    },
    "newColumnName": "acceptedScientificName",
    "columnInsertIndex": 2,
    "baseColumnName": "callAPIworms",
    "expression": "grel:value.parseJson().get('valid_name')",
    "onError": "set-to-blank"
  },
  {
    "op": "core/column-move",
    "description": "Move column scientificName to position 2",
    "columnName": "scientificName",
    "index": 2
  },
  {
    "op": "core/column-addition",
    "description": "Create column scientificNameIDSuggested at index 2 based on column callAPIworms",
    "engineConfig": {
      "mode": "row-based",
      "facets": []
    },
    "newColumnName": "scientificNameIDSuggested",
    "columnInsertIndex": 2,
    "baseColumnName": "callAPIworms",
    "expression": "grel:value.parseJson().get('lsid')",
    "onError": "set-to-blank"
  },
  {
    "op": "core/column-addition",
    "description": "Create column authorshipValidation at index 2 based on column scientificNameIDSuggested",
    "engineConfig": {
      "mode": "row-based",
      "facets": []
    },
    "newColumnName": "scientificNameIDValidation",
    "columnInsertIndex":2,
    "baseColumnName": "scientificNameIDSuggested",
    "expression": "grel:if(value==cells[\"scientificNameID\"].value,'1','0').toString()",
    "onError": "set-to-blank"
  },
  {
    "op": "core/column-move",
    "description": "Move column scientificNameID to position 2",
    "columnName": "scientificNameID",
    "index": 2
  },
  {
    "op": "core/column-addition",
    "description": "Create column A at index 2 based on column callAPIworms",
    "engineConfig": {
      "facets": [],
      "mode": "row-based"
    },
    "newColumnName": "taxonMatchType",
    "columnInsertIndex": 2,
    "baseColumnName": "callAPIworms",
    "expression": "grel:value.parseJson().get('match_type')",
    "onError": "set-to-blank"
  }
]