update file names

Author: RyanJField

content/docs/api/_index.md

@@ -0,0 +1,165 @@
+---
+weight: 4
+title: "Modelling APIs"
+bookCollapseSection: true
+---
+
+# API
+
+The API is defined more in terms of file formats than it is in terms of data types. There are two file formats that are native to the data pipeline, and files in these formats are referred to as *data products*: TOML files, and HDF5 files. TOML files store “small” parameter data, representing individual parameters. HDF5 files are used to store structured data, encoded either as “arrays” or “tables”. Both formats are described in more detail below, alongside API functions used to interact with them. Data in any other file format are treated as binary blobs, and are referred to as *external objects*.
+
+Different metadata is stored about each -- data products record information about their internal structure and naming of their components, whereas external objects record information about their provenance (since data products are internal to the pipeline, provenance is recorded separately). A single object can be both an external object and a data product, and thus have both sets of metadata recorded.
+
+## Initialisation
+
+The API must be initialised with the model URI and git sha, which should then be set as run-level metadata.
+
+## Additional metadata on write
+
+The write functions all accept `description` and `issues` arguments.
+
+## TOML (parameter) files
+
+A parameter file contains representations of one or more parameters, each a single number, possibly with some associated uncertainty. Parameters may by represented as point-estimates, parametric distributions, and sample data.
+
+### File format
+
+Parameters are stored in toml-formatted files, with the extension “toml”, containing sections corresponding to different components. The following is an example of the internal encoding, defining three components: "`my-point-estimate`", "`my-distribution`", and "`my-samples`":
+
+```
+[my-point-estimate] 
+type = "point-estimate" 
+value = 0.1 
+
+[my-distribution] 
+type = "distribution" 
+distribution = "gamma" 
+shape = 1 
+scale = 2 
+ 
+[my-samples] 
+type = "samples" 
+samples = [1.0, 2.0, 3.0, 4.0, 5.0] 
+```
+
+Point estimates are used when our knowledge of the parameter is only sufficient for a single value, with no notion of uncertainty. A point estimate component must have type = "point-estimate" and a value that is either a float or an integer.
+
+Distributions are used when our knowledge of a parameter can be represented by a parametric distribution. A distribution component must have type = "distribution", a distribution set to a string name of the distribution, and other parameters determined by the distribution. The set of distributions required to be supported is currently undefined.
+
+Samples are used when our knowledge of a parameter is represented by samples, from either empirical measurements, or a posterior distribution. A samples component must have type = "samples" and a value that is a list of floats and integers.
+
+#### Distributions
+
+The supported distributions,each with a link to information about their parameterisation, and their standardised parameter names are as follows:
+
+
+| Distribution               | Standardised parameter names               |
+| -------------------------- | ------------------------------------------ |
+| categorical (non-standard) | bins (string array), weights (float array) |
+| gamma                      | k (float), theta (float)                   |
+| normal                     | mu (float), sigma (float)                  |
+| uniform                    | a (float), b (float)                       |
+| poisson                    | lambda (float)                             |
+| exponential                | lambda (float)                             |
+| beta                       | alpha (float), beta (float)                |
+| binomial                   | n (int), p (float)                         |
+| multinomial                | n (int), p (float array)                   |
+
+### API functions
+
+`read_estimate(data_product, component) -> float or integer`
+
+If the component is represented as a point estimate, return that value.
+
+If the component is represented as a distribution, return the distribution mean.
+
+If the component is represented as samples, return the sample mean.
+
+`read_distribution(data_product, component) -> distribution object`
+
+If the component is represented as a point estimate, fail.
+
+If the component is represented as a distribution, return an object representing that distribution.
+
+If the component is represented as samples, failreturn an empirical distribution.
+
+`read_samples(data_product, component) -> list of floats or integers`
+
+If the component is represented as a point estimate, fail.
+
+If the component is represented as a distribution, fail.
+
+If the component is represented as samples, return the samples.
+
+`write_estimate(data_product, component, estimate, description, issues)`
+
+`write_distribution(data_product, component, distribution object, description, issues)`
+
+`write_samples(data_product, component, samples, description, issues)`
+
+## HDF5 files
+
+<span style="font-size:12pt; color:red">Note that the following is subject to change. For example, we may want to add all of the metadata as attributes.</span>
+
+An HDF5 file can be either a table or an array. A table is always 2-dimentional and might typically be used when each column contains different classes of data (*e.g.* integers and strings). Conversely, all elements in an array should be the same class, though the array itself might be 1-dimensional, 2-dimensional, or more (*e.g.* a 3-dimensional array comprising population counts, with rows as area, columns as age, and a third dimension representing gender).
+
+You should create a single HDF5 file for a single dataset. Unless you have a dataset that really should have been generated as multiple datasets in the first place (*e.g.* testing data mixed with carehome data), in which case use your own judgement.
+
+HDF5 files contain structured data, encoded as either an “array”, or a “table”, both of which are described in more detail below.
+
+### File format
+
+HDF5 files are stored with the extension “h5”. Internally, each component is stored in a different (possibly nested) group, where the full path defines the component name (*e.g.* “path/to/component”). Inside the group for each component is either a value named “array”, or a value named “table”. It is an error for there to be both.
+
+#### array format
+
+{component}/array
+: An n-dimensional array of numerical data
+
+{component}/Dimension_{i}_title
+: The string name of dimension {{< katex >}}i{{< /katex >}}
+
+{component}/Dimension_{i}_names
+: String labels for dimension {{< katex >}}i{{< /katex >}}
+
+{component}/Dimension_{i}_values
+: Values for dimension {{< katex >}}i{{< /katex >}}
+
+{component}/Dimension_{i}_units  
+: Units for dimension {{< katex >}}i{{< /katex >}}
+
+{component}/units
+: Units for the data in array
+
+#### table format
+
+{component}/table
+: A dataframe
+
+{component}/row_names
+: String labels for the row axis
+
+{component}/column_units
+: Units for the columns
+
+### API functions
+
+`read_array(data_product, component) -> array`
+
+If the component does not terminate in an array-formatted value, raise an error.
+
+Return an array, currently with no structural information.
+
+`read_table(data_product, component) -> dataframe`
+
+If the component does not terminate in a table-formatted value, raise an error.
+
+Return a dataframe, with labelled columns.
+
+`write_array(data_product, component, array, description, issues)`
+
+If the array argument is not array-formatted, raise an error.
+
+`write_table(data_product, component, table, description, issues)`
+
+If the table argument is not table-formatted, raise an error.

content/docs/api/c/_index.md

@@ -0,0 +1,5 @@
+---
+weight: 5
+title: "C"
+bookCollapseSection: false
+---

content/docs/api/cpp/_index.md

@@ -0,0 +1,29 @@
+---
+weight: 6
+title: "C++"
+bookCollapseSection: false
+---
+
+# cppDataPipeline
+
+The `cppDataPipeline` library contains functions used to interface with the FAIR Data Pipeline in C++.
+
+## Building
+
+The library can be built using cmake and requires a C++ 11 compatible compiler
+
+```
+cmake -Bbuild
+cmake --build build
+```
+
+## Usage
+
+See the [package documentation][docs] for instructions and examples.
+
+## Source code
+
+See the package's [code repo][repo].
+
+[docs]: https://www.fairdatapipeline.org/cppDataPipeline/
+[repo]: https://github.com/FAIRDataPipeline/cppDataPipeline

content/docs/api/fortran/_index.md

@@ -0,0 +1,5 @@
+---
+weight: 7
+title: "FORTRAN"
+bookCollapseSection: false
+---

content/docs/api/java/_index.md

@@ -0,0 +1,60 @@
+---
+weight: 4
+title: "Java"
+bookCollapseSection: true
+---
+
+# javaDataPipeline
+
+The javaDataPipeline Java library allows Java modellers to interface with the FAIR Data Pipeline.
+It is available on Maven Central:
+
+```gradle
+dependencies {
+    implementation 'org.fairdatapipeline:api:1.0.0-beta'
+}
+```
+
+The main purpose of interfacing with the FAIR Data Pipeline is the recording of Coderuns: a Coderun is a recorded session of your modelling code, all the inputs used and outputs generated will be recorded and a [Provenance Report][prov] will be generated.
+
+The basic idea behind the FAIR DataPipeline and Coderuns is explained on the [Data pipeline](/docs/interface/) page.
+
+
+
+
+
+
+
+
+## Usage
+
+A useful starting point to get to know the javaDataPipeline is to explore the [java Simple Model](https://github.com/FAIRDataPipeline/javaSimpleModel). This shows example [config.yaml](https://www.fairdatapipeline.org/docs/interface/config/) files, example
+`main(String[] args)` code that allows `fair run` to call your code, and example `Coderun()` code for reading/writing input/output Data Products.
+
+
+The links below give a basic introduction to the javaDataPipeline; these should be read in conjunction with the [javaDocs][docs] which give more precise specification of the java API.
+
+[Basic Coderun](coderun)  
+[Issues](issues)   
+[Parameters](parameters)  
+[HDF5](hdf5)   
+[Run](run)  
+[Debug](debug)  
+
+See the [package documentation][docs] for Javadoc.
+
+
+## License/copyright
+
+Copyright (c) 2021: Bram Boskamp, Biomathematics and Statistics Scotland and the Scottish COVID-19 Response Consortium
+
+GNU GENERAL PUBLIC LICENSE Version 3
+
+
+## Source code
+
+See the package's [code repo][repo].
+
+[docs]: https://www.fairdatapipeline.org/javaDataPipeline
+[repo]: https://github.com/FAIRDataPipeline/javaDataPipeline
+[prov]: /docs/data_registry/prov_report/

content/docs/api/java/coderun.md

@@ -0,0 +1,57 @@
+---
+weight: 10
+---
+
+# javaDataPipeline Coderun
+
+The main class used to interface with the FAIR DataPipeline in Java, is the `Coderun` class.
+
+Users should initialise the `Coderun` instance using a *try-with-resources* block or ensure that `.close()` is explicitly called upon finishing.
+
+The `Coderun` constructor needs a Path to the config.yaml, a Path to the script.sh, and the registry authentication token.
+
+The user then would access the input data product(s) using `coderun.get_dp_for_read(dataproduct_name)`
+and output data product(s) using `coderun.get_dp_for_write(dataproduct_name, extension)`.
+
+From the resulting data_product we can then access either one unnamed object_component: `Object_component_read oc = dp.getComponent()`
+or any number of named object_components: `Object_component_read oc1 = dp.getComponent('ComponentName')`.
+
+
+Data in FAIR Data Pipeline internal formats (HDF5, TOML) is written to (or read from) named `Object_components`, while other file formats (such as CSV) are written to (or read from) just 1 unnamed `Object_component`.
+
+Here is an example of a `Coderun` writing a CSV file to one unnamed `Object_component`:
+
+```
+    try (var coderun = new Coderun(configPath, scriptPath, token)) {
+      Data_product_write dp = coderun.get_dp_for_write(dataProduct, "csv");
+      Object_component_write oc = dp.getComponent();
+      Path p = oc.writeLink();
+      write_my_csv_data_to_path(p);
+    }
+```
+
+And for reading:
+
+```
+    try (var coderun = new Coderun(configPath, scriptPath, token)) {
+      Data_product_read dp = coderun.get_dp_for_read(dataProduct);
+      Object_component_read oc = dp.getComponent();
+      Path p = oc.readLink();
+      read_my_csv_data_from_path(p);
+    }
+```
+
+
+Here is an example of a coderun writing Samples to one named Object_component:
+
+```Java
+    try (var coderun = new Coderun(configPath, scriptPath)) {
+       ImmutableSamples samples = ImmutableSamples.builder().addSamples(1, 2, 3).rng(rng).build();
+       String dataProduct = "animal/dodo";
+       String component1 = "example-samples-dodo1";
+       Data_product_write dp = coderun.get_dp_for_write(dataProduct, "toml");
+       Object_component_write oc1 = dp.getComponent(component1);
+       oc1.writeSamples(samples);
+    }
+```
+

content/docs/api/java/debug.md

@@ -0,0 +1,37 @@
+---
+weight: 60
+---
+
+# Debugging / troubleshooting
+
+## Logger
+
+The javaDataPipeline library uses *SLF4J* (Simple Logging Facade for Java).
+
+You can either use this with the SLF4J simple logger, or it can bind to your own favourite logging framework.
+In order to set the default simple logger to more verbose logging:
+
+- add the `org.slf4j:slf4j-simple` dependency
+- set the log level:
+
+```System.setProperty(org.slf4j.impl.SimpleLogger.DEFAULT_LOG_LEVEL_KEY, "TRACE");```
+
+## Running manually
+
+Instead of running:
+
+```
+fair run src/main/resources/config.yaml
+```
+
+You can call `fair run` and your actual java code separately:
+
+```
+fair run --ci src/main/resources/config.yaml
+gradle run --args "/tmp/tmpXXXXXXX/data_store/jobs/<timestamp>"
+```
+
+The actual tmp location is shown as output from the `fair run` command.
+
+This may help debug/troubleshoot.
+

content/docs/api/java/hdf5.md

@@ -0,0 +1,7 @@
+---
+weight: 40
+---
+
+# HDF5
+
+HDF files are not supported in the current version of the Java FAIR Data Pipeline API.