Repository navigation
Example process
For the purposes of this document we have selected an example based on the Painting Conservation Glossary from the Smithsonian Conservation Institute available here.
Note: the URIs used in this example have not been registered with the w3id.org repository and are only here for the purposes of the example.
The following Python script performs web scraping - extracting terms and descriptions from the list on the Smithsonian Painting Conservation Glossary webpage:
import requests
from lxml import html
# replace tabs with spaces, normalize and trim whitespace
def clean(s):
return " ".join(str(s).replace("\t", " ").strip().split())
# get the HTML page content from web URL
PAGE_URL = "https://www.si.edu/mci/english/learn_more/taking_care/painting_glossary.html"
LOCAL_FILE = "smithsonian.csv"
page = requests.get(PAGE_URL, timeout=5.000)
uri_base = "https://w3id.org/spg/concept/"
# success?
if page.status_code == 200:
# parse out the specific list items we are interested in
tree = html.fromstring(page.content)
items = tree.xpath('//div[@id="site_sections"]/ul/li/p')
with open(LOCAL_FILE, "w", encoding="utf-8") as output:
line = f"\"concept id\",\"concept\",\"description\"\n" # header
output.write(line)
i = 0
for item in items:
i = i + 1
# parse and clean terms and descriptions
concept = clean(item.xpath('./strong/text()')[0])
desc = clean(item.xpath('./text()')[0][2:]) # removed first 2 chars (colon and space)
desc = desc.replace('"','""') # escape quotation marks in the description
line = f"\"{uri_base}{i}\",\"{concept}\",\"{desc}\"\n" # item formatted as comma delimited term and description
output.write(line)
else:
print(f"Could not get data, status code {page.status_code} returned")The script extracts the list of terms and descriptions and writes the results to a local CSV file which can be found here.
The above list includes terms which refer to more than one concepts, such as https://w3id.org/spg/concept/29 glaze and https://w3id.org/spg/concept/41 light fastness. In the next step we manually split these terms into separate concepts.
| URI | label | scope note |
|---|---|---|
| https://w3id.org/spg/concept/86 | glaze (glass-like surface production) | To impart a glass-like surface. Aged glaze is very sensitive to solvents. |
| https://w3id.org/spg/concept/87 | light fastness (dimension) | The relative degree of change or lack of change in color of materials exposed to the same amount and character of light. |
| https://w3id.org/spg/concept/29 | glaze (coloring) | To cover paler under painting with a layer consisting of transparent pigments and excess medium. Traditionally used to add color to forms modeled in monochrome opaque paint. |
| https://w3id.org/spg/concept/41 | light fastness (color quality) | ability to withstand color changes on exposure to light. |
The resulting CSV file can be found here
The CSV file is then re-formatted and saved as .xlsx based on the template provided by the SKOS Play website. The file can be found here. Notice that in this file concepts have been marked as broader to other concepts. Related links have also been established.
The SKOS Play conversion results in a Turtle file as shown here.
The Turtle file from the last step can be packaged with a version for uploading to the repository. The complete Turtle file can be seen online here: spg-20210325.trig.
A similar .xlsx file is used to match vocabulary concepts to the AAT. This is done manually by domain experts, i.e. in this case a conservator with relevant experience. This template file does not include scope notes, broader concepts or related concepts. Instead it includes the required SKOS properties for matching. The file can be seen here.
The SKOS Play conversion results in a Turtle file as shown here.
The Turtle file from the last step can be packaged with a version for uploading to the repository. The complete Turtle file can be seen online here: conservation-vocabularies/spg-20210325--aat-20200518.trig