Skip to content

Commit 9f48e9a

Browse files
author
irinaschubert
authored
feat(excel-to-properties): create properties from Excel (DSP-1577) (#89)
* Squashed commit of the following: commit e24a895 Author: irinaschubert <irina.schubert@dasch.swiss> Date: Thu Sep 9 15:49:19 2021 +0200 add resources module to bazel commit 0aefee9 Author: irinaschubert <irina.schubert@dasch.swiss> Date: Thu Sep 9 15:44:29 2021 +0200 Update dsp-tools-usage.md commit 3ddecdc Author: irinaschubert <irina.schubert@dasch.swiss> Date: Thu Sep 9 15:41:26 2021 +0200 add documentation commit 914d3a6 Author: irinaschubert <irina.schubert@dasch.swiss> Date: Thu Sep 9 15:31:36 2021 +0200 update test commit 75a04ae Author: irinaschubert <irina.schubert@dasch.swiss> Date: Thu Sep 9 15:11:07 2021 +0200 add test commit 1fb39b9 Author: irinaschubert <irina.schubert@dasch.swiss> Date: Thu Sep 9 14:40:17 2021 +0200 add validation and schema commit 6275c13 Author: irinaschubert <irina.schubert@dasch.swiss> Date: Thu Sep 9 12:51:20 2021 +0200 integrate script into dsp-tools * integrate standalone script from dsp-tools-prep into dsp-tools * add documentation * update documentation
1 parent 7b0302f commit 9f48e9a

13 files changed

Lines changed: 348 additions & 5 deletions

‎MANIFEST.in‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,5 +3,6 @@ include knora/dsplib/utils/knora-schema.json
33
include knora/dsplib/utils/knora-schema-lists.json
44
include knora/dsplib/utils/knora-schema-lists-only.json
55
include knora/dsplib/utils/knora-schema-resources-only.json
6+
include knora/dsplib/utils/knora-schema-properties-only.json
67
include knora/dsplib/utils/knora-data-schema.xsd
78
include knora/dsplib/utils/language-codes-3b2_csv.csv
724 KB
Loading

‎docs/dsp-tools-excel.md‎

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,23 @@ For further information about resources, see [here](./dsp-tools-create.md#resour
2525

2626
## Create the properties for a data model from an Excel file
2727

28-
[not yet implemented]
28+
With dsp-tools the `properties` section used in a data model (JSON) can be created from an Excel file. Only the first worksheet of
29+
the Excel file is considered and only XLSX files are allowed. The `properties` section can be inserted into the ontology file and
30+
then be uploaded onto a DSP server.
31+
32+
The Excel sheet must have the following format:
33+
![img-properties-example.png](assets/images/img-properties-example.png)
34+
35+
The expected columns are:
36+
37+
- `name` : The name of the property
38+
- `super` : The base property of the property
39+
- `object` : The resource the property refers to if it is a link property (property derived from `hasLinkTo`)
40+
- `en`, `de`, `fr`, `it` : The labels of the property in different languages, at least one language has to be provided
41+
- `gui_element` : The GUI element for the property
42+
- `hlist` : In case of list values the according list
43+
44+
For further information about properties, see [here](./dsp-tools-create.md#properties).
2945

3046
## Create a DSP-conform XML file from an Excel file
3147

‎docs/dsp-tools-usage.md‎

Lines changed: 23 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -125,15 +125,36 @@ the usage of this command can be found [here](./dsp-tools-excel.md#create-a-list
125125
dsp-tools excel2resources excel_file.xlsx output_file.json
126126
```
127127

128-
The command is used to create the resource section of an ontology from an Excel file. Therefore, an Excel file has to be provided
128+
The command is used to create the resources section of an ontology from an Excel file. Therefore, an Excel file has to be provided
129129
with the data in the first worksheet of the Excel file.
130130

131-
The following example shows how to create the resources section from an Excel file called `Resources.xlsx`.
131+
The following example shows how to create the resources section from an Excel file called `Resources.xlsx`. The output is written
132+
to a file called `resources.json`.
132133

133134
```bash
134135
dsp-tools excel2resources Resources.xlsx resources.json
135136
```
136137

137138
More information about the usage of this command can be
138139
found [here](./dsp-tools-excel.md#create-the-resources-for-a-data-model-from-an-excel-file)
140+
.
141+
142+
## Create properties from an Excel file
143+
144+
```bash
145+
dsp-tools excel2properties excel_file.xlsx output_file.json
146+
```
147+
148+
The command is used to create the properties section of an ontology from an Excel file. Therefore, an Excel file has to be
149+
provided with the data in the first worksheet of the Excel file.
150+
151+
The following example shows how to create the properties section from an Excel file called `Properties.xlsx`. The output is
152+
written to a file called `properties.json`.
153+
154+
```bash
155+
dsp-tools excel2properties Properties.xlsx properties.json
156+
```
157+
158+
More information about the usage of this command can be found
159+
[here](./dsp-tools-excel.md#create-the-properties-for-a-data-model-from-an-excel-file)
139160
.

‎docs/index.md‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,5 +24,9 @@ dsp-tools helps you with the following tasks:
2424
creates a JSON or XML file from one or several Excel files. The created data can either be integrated into an ontology or be
2525
uploaded directly to a DSP server with `dsp-tools create`.
2626
- [`dsp-tools excel2resources`](./dsp-tools-usage.md#create-resources-from-an-excel-file)
27-
creates the ontology's resource section from an Excel file. The resources can be integrated into an ontology and then be
27+
creates the ontology's resource section from an Excel file. The resulting section can be integrated into an ontology and then be
2828
uploaded to a DSP server with `dsp-tools create`.
29+
- [`dsp-tools excel2properties`](./dsp-tools-usage.md#create-properties-from-an-excel-file)
30+
creates the ontology's properties section from an Excel file. The resulting section can be integrated into an ontology and then
31+
be uploaded to a DSP server with `dsp-tools create`.
32+

‎knora/dsp_tools.py‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@
1515
from dsplib.utils.onto_get import get_ontology
1616
from dsplib.utils.excel_to_json_lists import list_excel2json, validate_list_with_schema
1717
from dsplib.utils.excel_to_json_resources import resources_excel2json
18+
from dsplib.utils.excel_to_json_properties import properties_excel2json
1819
from dsplib.utils.onto_validate import validate_ontology
1920
from dsplib.utils.xml_upload import xml_upload
2021

@@ -93,6 +94,14 @@ def program(args: list) -> None:
9394
parser_excel_resources.add_argument('outfile', help='Path to the output JSON file containing the resource data',
9495
default='resources.json')
9596

97+
parser_excel_properties = subparsers.add_parser('excel2properties', help='Create a JSON file from an Excel file containing '
98+
'properties for a DSP ontology. ')
99+
parser_excel_properties.set_defaults(action='excel2properties')
100+
parser_excel_properties.add_argument('excelfile', help='Path to the Excel file containing the properties',
101+
default='properties.xlsx')
102+
parser_excel_properties.add_argument('outfile', help='Path to the output JSON file containing the properties data',
103+
default='properties.json')
104+
96105
args = parser.parse_args(args)
97106

98107
if not hasattr(args, 'action'):
@@ -145,6 +154,9 @@ def program(args: list) -> None:
145154
elif args.action == 'excel2resources':
146155
resources_excel2json(excelfile=args.excelfile,
147156
outfile=args.outfile)
157+
elif args.action == 'excel2properties':
158+
properties_excel2json(excelfile=args.excelfile,
159+
outfile=args.outfile)
148160

149161

150162
def main():

‎knora/dsplib/utils/BUILD.bazel‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,16 @@ py_library(
2424
]
2525
)
2626

27+
py_library(
28+
name = "excel_to_json_properties",
29+
visibility = ["//visibility:public"],
30+
srcs = ["excel_to_json_properties.py"],
31+
deps = [
32+
requirement("jsonschema"),
33+
requirement("openpyxl")
34+
]
35+
)
36+
2737
py_library(
2838
name = "expand_all_lists",
2939
visibility = ["//visibility:public"],
Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
import json
2+
import os
3+
4+
import jsonschema
5+
from openpyxl import load_workbook
6+
7+
8+
def validate_properties_with_schema(json_file: str) -> bool:
9+
"""
10+
This function checks if the json properties are valid according to the schema.
11+
12+
Args:
13+
json_file: the json with the properties to be validated
14+
15+
Returns:
16+
True if the data passed validation, False otherwise
17+
18+
"""
19+
current_dir = os.path.dirname(os.path.realpath(__file__))
20+
with open(os.path.join(current_dir, 'knora-schema-properties-only.json')) as schema:
21+
properties_schema = json.load(schema)
22+
23+
try:
24+
jsonschema.validate(instance=json_file, schema=properties_schema)
25+
except jsonschema.exceptions.ValidationError as err:
26+
print(err)
27+
return False
28+
print('Properties data passed schema validation.')
29+
return True
30+
31+
32+
def properties_excel2json(excelfile: str, outfile: str):
33+
"""
34+
Converts properties described in an Excel file into a properties section which can be integrated into a DSP ontology
35+
36+
Args:
37+
excelfile: path to the Excel file containing the properties
38+
outfile: path to the output JSON file containing the properties section for the ontology
39+
40+
Returns:
41+
None
42+
"""
43+
# load file
44+
wb = load_workbook(filename=excelfile, read_only=True)
45+
sheet = wb.worksheets[0]
46+
props = [row_to_prop(row) for row in sheet.iter_rows(min_row=2, values_only=True, max_col=9)]
47+
48+
prefix = '"properties":'
49+
50+
if validate_properties_with_schema(json.loads(json.dumps(props, indent=4))):
51+
# write final list to JSON file if list passed validation
52+
with open(file=outfile, mode='w+', encoding='utf-8') as file:
53+
file.write(prefix)
54+
json.dump(props, file, indent=4)
55+
print('Properties file was created successfully and written to file:', outfile)
56+
else:
57+
print('Properties data is not valid according to schema.')
58+
59+
return props
60+
61+
62+
def row_to_prop(row):
63+
"""
64+
Parses the row of an Excel sheet and makes a property from it
65+
66+
Args:
67+
row: the row of an Excel sheet
68+
69+
Returns:
70+
prop (JSON): the property in JSON format
71+
"""
72+
name, super_, object_, en, de, fr, it, gui_element, hlist = row
73+
labels = {}
74+
if en:
75+
labels['en'] = en
76+
if de:
77+
labels['de'] = de
78+
if fr:
79+
labels['fr'] = fr
80+
if it:
81+
labels['it'] = it
82+
if not labels:
83+
raise Exception(f"No label given in any of the four languages: {name}")
84+
prop = {
85+
'name': name,
86+
'super': [super_],
87+
'object': object_,
88+
'labels': labels,
89+
'gui_element': gui_element
90+
}
91+
if hlist:
92+
prop['gui_attributes'] = {'hlist': hlist}
93+
return prop

0 commit comments

Comments
 (0)