Code template and doc for Substitutions #25
Comments
|
The mock-up contains a functioning separation between data access, I will build a template for Laney and rewrite the documentation, so you |
MichaelErichsen
changed the title
|
Hi Michael and Laney,
I am pleased that Michael thinks that my proposal is a good idea!
I am meeting Rod and Don in Melbourne at the end of next (21 and 22
June) to review the priority of tasks here. Don and I are also meeting
some anthropologist researchers working in a first nation law firm. They
use TMG as do some other similar researchers in other companies in that
field.
One of the items on the agenda is to discuss what external data (not in
the HRE H2 database) needs to be be accessible to the substitution
templates. Currently I have listed 3 files for the Client and 3 similar
but different file for the Server. They are (labels for convenience of
discussion):
CCE – the Client Common Environment local file (XML) Installation data
and status
CUE– the Client Users Environment file (XML) User population properties
and status
CPE – the Client Projects Environment file (XML) Project external
properties and status of use
SCE – the Server Common Environment (XML) Installation data and status
SUE– the Server Users Environment file (XML) User population properties
and status
SPE – the Server Projects Environment file (XML) Project external
properties and status of use
We wanted to start to identify what data needs to be held in these 6
files and then the operations that would be needed to maintain these
environment file and hence an API definition for each.
No doubt Michael will also know of other data that we have not considered.
Robin
…On 13-Jun-18 03:29 PM, Michael Erichsen wrote:
The mock-up contains a functioning separation between data access,
business logic, communication logic, and user interface, which is
already implementing both running client/server and single-machine,
implementing the thoughts of Nils, and which have been tested by him. It
will of course need refinement along the way.
I will build a template for Laney and rewrite the documentation, so you
can write your part to be plugged into the skeleton. If I am lucky it
can be done in about a week's time.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#25 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AVeLtE5rmYttDRdUuqZUcKMU13zcUTU9ks5t8KpDgaJpZM4UllEv>.
--
Robin Lamacraft, Adelaide, Australia
---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus
|
|
Just a general comment on data formats, which in and around HRE could be XML, JSON, H2, CSV, unstructured, or Java/Eclipse preference files. We have earlier discussed preferring JSON to XML as the trend has clearly gone this way. For properties/preferences in the HRE application the internal representation is a properties/preferences file, which is maintained by the Eclipse platform. From the user side it is maintained in Preferences pages. So the operations and API is already provided by the platform: http://www.myerichsen.net/HRE/Shot1.png Definitions to be imported into HRE could be any format appropriate to other systems producing them. That could be a installation configuration file, perhaps? This what the HRE skeleton produces right now: CSMODE=DIRECT The preferences are persisted by Eclipse in <HRE_home>\workspace.metadata.plugins\org.eclipse.core.runtime.settings\org.historyresearchenvironment.client.prefs and are not expected to be read or written by the user. Michael |
|
Hi Michael and Laney,
There are several issues that I am working through related to the
Substitutions area:
(1) What style and sets of parameters are needed in the APIs to initiate
the execution of a Substitution, so that each API is targeted to a
specific Substitution outcome.
(2) Documenting the different outcome types of the Substitutions area
with examples.
(3) What kind of user GUI and associated BR APIs should be available to
create or modify a Substitution definition.
(4) How to store the suggested 6(?) external environment files? I agree
that JSON looks like a more straight forward and compact representation.
[NOTE 1:The JSON parser has a value length limit of 8192 characters and
this will be OK for some uses, but not all. The 4MB max file size of
JSON would be insufficient for some uses, but would suitable for these
external environment files. JSON's parser has 8 restricted keyboard
characters, also Unicode characters that need backslash eliding.
Management of those characters for data environment storage is an
overhead as that eliding is not required within the HRE database
strings. (Should it?) ]
There are similar issues when using XML, so where the constraints of
field length and total files size are within the JSON constraints then
JSON would be preferable.
[NOTE 2: It is likely for bulk data importing and exporting that XML
would be the preferred format when it is likely that the processing may
exceed the JSON limits. Hence HRE will need a parser and an editor for
both JSON and XML formats in the released product.]
(5) Documenting what data values should be managed and retrievable from
the 6 external environment files.
(6) identifying what (if any) in the Eclipse application resource file
may need to be accessed by substitution (or cached in a user readable space)
Regards,
Robin
…On 13-Jun-18 03:28 PM, Michael Erichsen wrote:
Robin:
here would a need to implement an agreed set of APIs to the
Substitution area and they would used in the skeleton a crude
placeholder module was included. That would be replaced by the full
one once it had been reasonably test in its sandpit. The Substitutions
area would rely on another agreed set of APIs, to built by Michael to
prove the storage of the template elementes and the database retrieval
of values that Laney's code wanted to use, but at the start that can
be ghosted in a sandpit.
THE PROPOSED TASK FOR LANEY (Overview - more detail later if agreed)
In TMG there are namestyle templates and sentences templates where a
markup means to substitute that mark with a user data derived text
string. The same concept applies in HRE with a more complete set of
operators and larger environment from which retrieved values may be
sourced. In HRE, the use of this mechanism is extended to error and
warning messages, filter definitions, sorting definitions, memos with
embedded Citations, selection of columns for tabular output and later
in other types of report generation.
In HRE jargon this is the Substitutions area. The definition of and
the evaluation of substitution templates becomes the core that
provides the flexibility for the user to configure HRE to match their
workflow needs. The substitutions area directly involves about 14
database tables and some auxiliary tables for each use. case.
The task would involve the GUI to create/edit/store/evaluate the
template an the code required to manage the associated database tables
(via APTs defined with Michael). In the process Business Rules level
would need to provide validity checks for definition editing and
execution.
Much of the preloading of the lookup tables in the database and the
ability to store user data for any manual or importing from TMG will
rely their being agreed structures and methods from the substitution area.
INTERACTIONS with the team
MICHAEL
(1) to continue with the structure of skeleton including GUI and other
BR modules as required
(2) with LANEY jointly to define APIs for evaluation of templates for
each type of use of Substitutions
(3) with Laney jointly defined APIs for the storage and retrieval of
data related to definition and use of Substitutions.
LANEY
(1) to build the GUI for defining/editing/managing/validating any use
of Substitutions
(2) code to evaluate substitution templates and return the content to
the requester
ROBIN
(1) to provide additional documentation on the internals of
Substitutions and their use cases.
(2) This documentation is currently being drafted to augment what has
been already placed on GitHub
(3) To liaise with MICHAEL and LANEY to ensure the intention of the
data model is achieved. It is quite likely that minor changes in the
field set of some tables may need adjustment.
INTEGRATION
When LANEY's code has be tested in isolation and seems robust enough,
then it can be integrated into the HRE skeleton.
There are other important tasks,
(a). dates and time intervals (Java),
(b). field and record validation rules (Jython)
that can start in the same way if others wish to be come involved.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#25>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AVeLtNo7xRIpUN64YIa6qK_QLQ1u_HCUks5t8KoegaJpZM4UllEv>.
--
Robin Lamacraft, Adelaide, Australia
---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus
|
|
Hi Robin I am not aware of such a size limitation in JSON. Is there anything which I have failed to see? The JSON support in the mock up is coded using org.json Java classes, and I don't find any limitations there. Br. Michael |
|
Hi Michael,
I did search the Internet and found some comparisons of JSON and XML
that implied that there were some constraints on JSON parsing. I would
be pleased if the JSON field value maximum length was some what greater
than 8192 characters and the total JSON file size was greater then 4MB.
All good - may be these sites were either out of date or only applied
some particular implementations. I am just being couscous - it is better
to find such things out before one commits to them.
The extreme use of one of these (JSON or XML) formats in HRE would be to
dump an entire very large HRE H2 database as a machine-readable form of
application-neutral archive backup file.
Robin
…On 14-Jun-18 03:14 PM, Michael Erichsen wrote:
Hi Robin
I am not aware of such a size limitation in JSON. Is there anything
which I have failed to see? The JSON support in the mock up is coded
using org.json Java classes, and I don't find any limitations there.
Br. Michael
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
<#25 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AVeLtAKGzqA08w_qVGqzQos1zA0XWhZTks5t8fhVgaJpZM4UllEv>.
--
Robin Lamacraft, Adelaide, Australia
---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus
|
|
Hi Robin For this purpose I would choose .csv (comma separated values). First, it is directly importable in spreadsheets and other databases, and secondly, H2 includes native functionality to import and export csv. I have already implemented that in the table csv import and export functionality in the DBADMIN and now in the skeleton. final Csv csvFile = new Csv(); and final H2TableProvider provider = new H2TableProvider(tableName); Br Michael |
|
Hi Michael,
If you are suggesting the use of CSV format for the 6 environment
property and status files then this may be a problem because I was
looking a format that could easily allows for up to 3 levels of nested
information to reduce duplication.
I know that this could be overcome by using more files, one for each
cross-product between pairs of controlling margins. But that would lead
to potentially more consistency checking as compared with the more
normalized nested structure.
Robin
…On 14-Jun-18 05:01 PM, Michael Erichsen wrote:
Hi Robin
For this purpose I would choose .csv (comma separated values). First,
it is directly importable in spreadsheets and other databases, and
secondly, H2 includes native functionality to import and export csv. I
have already implemented that in the table csv import and export
functionality in the DBADMIN and now in the skeleton.
Code samples from
https://github.com/History-Research-Environment/HRE--History-Research-Environment/blob/develop/HRE--History-Research-Environment/org.historyresearchenvironment/bundles/org.historyresearchenvironment.client/src/org/historyresearchenvironment/databaseadmin/parts/H2TableNavigator.java:
final Csv csvFile = new Csv();
csvFile.setFieldSeparatorWrite(",");
csvFile.write(fileName, rs, "UTF-8");
and
final H2TableProvider provider = new H2TableProvider(tableName);
rowCount = provider.importCsv(fileName);
Br Michael
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
<#25 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AVeLtLq9VX5aahCMJ3q0I4qbu4pjTNYxks5t8hFKgaJpZM4UllEv>.
--
Robin Lamacraft, Adelaide, Australia
---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus
|
|
Hi Robin I only suggested csv for database and table import and export, using H2 features. Br , Michael |
|
Have started pulling the parts for a sample together. |
|
Class SampleView in package org.historyresearchenvironment.dataaccess generated from H2 VIEW using JPA. |
|
Finished and uploaded to Github. |
|
Will be uploaded as build 0.1.0.201807152107 |
|
Closed with build 0.1.0.201807160730. |
Robin:
here would a need to implement an agreed set of APIs to the Substitution area and they would used in the skeleton a crude placeholder module was included. That would be replaced by the full one once it had been reasonably test in its sandpit. The Substitutions area would rely on another agreed set of APIs, to built by Michael to prove the storage of the template elementes and the database retrieval of values that Laney's code wanted to use, but at the start that can be ghosted in a sandpit.
THE PROPOSED TASK FOR LANEY (Overview - more detail later if agreed)
In TMG there are namestyle templates and sentences templates where a markup means to substitute that mark with a user data derived text string. The same concept applies in HRE with a more complete set of operators and larger environment from which retrieved values may be sourced. In HRE, the use of this mechanism is extended to error and warning messages, filter definitions, sorting definitions, memos with embedded Citations, selection of columns for tabular output and later in other types of report generation.
In HRE jargon this is the Substitutions area. The definition of and the evaluation of substitution templates becomes the core that provides the flexibility for the user to configure HRE to match their workflow needs. The substitutions area directly involves about 14 database tables and some auxiliary tables for each use. case.
The task would involve the GUI to create/edit/store/evaluate the template an the code required to manage the associated database tables (via APTs defined with Michael). In the process Business Rules level would need to provide validity checks for definition editing and execution.
Much of the preloading of the lookup tables in the database and the ability to store user data for any manual or importing from TMG will rely their being agreed structures and methods from the substitution area.
INTERACTIONS with the team
MICHAEL
(1) to continue with the structure of skeleton including GUI and other BR modules as required
(2) with LANEY jointly to define APIs for evaluation of templates for each type of use of Substitutions
(3) with Laney jointly defined APIs for the storage and retrieval of data related to definition and use of Substitutions.
LANEY
(1) to build the GUI for defining/editing/managing/validating any use of Substitutions
(2) code to evaluate substitution templates and return the content to the requester
ROBIN
(1) to provide additional documentation on the internals of Substitutions and their use cases.
(2) This documentation is currently being drafted to augment what has been already placed on GitHub
(3) To liaise with MICHAEL and LANEY to ensure the intention of the data model is achieved. It is quite likely that minor changes in the field set of some tables may need adjustment.
INTEGRATION
When LANEY's code has be tested in isolation and seems robust enough, then it can be integrated into the HRE skeleton.
There are other important tasks,
(a). dates and time intervals (Java),
(b). field and record validation rules (Jython)
that can start in the same way if others wish to be come involved.
The text was updated successfully, but these errors were encountered: