Integration of DAT-CORE

[mguidon]

USE CASE

The file manager allows to select files from DAT-CORE and from SIM-CORE.
It only displays files that belong to the current user. Files are grouped by project name (SIM-CORE) and by datasets (DAT-CORE). I can search for files by name and by metadata.

I want to run the Clancy model by using the input file which is provided via the DAT-CORE. I want to store the raw data output file and one of the corresponding plots as a jpg.

USER STORY

• I login to Blackfynn, create a pair of secret keys.
• I upload the initial states file for the Clancy model (via Blackfynn).
• I create a user on Sim-Core and add the tokens to the user settings.
• Then I create the pipeline for executing the solver and the postprocessing.
• I attach a filemanager to the solver and as the input I can select the file which I have previously uploaded via Blackfynn.
• The postpro service has output ports to which I can attach file exporters.
• The file exporter allows me to upload the file locally or to Blackfynn (which I'm doing for a jpg file and for the raw output data).
• Now I go to the Blackfynn platform and have a look at my jpg file and at the data in a table view.

---

• Integrate DAT-CORE database using their API
• use for now to get input and store output parameters/data, potentially metadata and information about study (pipeline)
• tbd: user account management across COREs
• store Clancy-model output data on DAT_CORE
• pull, push and search functions are available

[mguidon]

Push pull and search (small and large data)

[mguidon]

@pcrespov as discussed, please populate

[pcrespov]

• Python client and API can be found here:
  • https://docs.blackfynn.io
  • https://developer.blackfynn.io
• Example of usage can be found in cchorn demo.
• Use test account provided to us. If you do not have one, pls contact Leonardo (via slack e.g.)

[mguidon]

• Have a look at Nexus (BBP)
• qx modeling storage

[mguidon]

Python API documentation can be found here:
http://docs.blackfynn.io/clients/python/quickstart.html

[drniiken]

• MaG

• some basic examples done

• makes some small blackfynn demo

• have py and rest interface to API in blackfynn

• generate plots and tables

• save file

• tagging of data is also possible

• or add various kinds of meta data

• see scripts provided by MaG

[pcrespov]

Canceron Milestone - dev notes

• Create a test that queries interface: search, pull and push. Add little sample code to upload/download
• Store Colin-Clancy in DAT-CORE together with their formation of what computational services and input used . this would allow us to hash things in a way that we know what changed or not

[drniiken]

• resolve issues of searching the Blackfynn platform
• integrate DAT-CORE as database backend for input-output of pipeline

[drniiken]

• MaG

• extended on last time demo

• created wrapper

• push pull works

• searching crashes server on their side

• think about how to use database

• same tech as they use, f3, have on in house server

• check with data standards people

• where simulations are stored

• we can store pp results in format which they can recognize

• customers should get a link to a file

• could be hosted by us or by them

• maybe put that in proposal

[mguidon]

Caprica Milestone - dev notes

• Get support for search issue