FAIRmat Users Meeting. A-B Session NOMAD for experimental data

[budschi]

FAIRmat Users Meeting A-B Session

I met @sanbrock today on planning the joint A-B session during users meeting. Please have a look and add your thoughts @aalbino2 @hampusnasstrom @Pepe-Marquez @RubelMozumder

Context

• from FAIRmat event website: On day two, we offer interactive sessions and hands-on workshops on
  research data management (RDM) and using NOMAD and NOMAD Oasis.

• Markus Session: https://events.fairmat-nfdi.eu/event/2/contributions/62/ :

  In this workshop we will cover the first steps with NOMAD and NOMAD Oasis.
  We will show you how to explore, upload, share and publish data with NOMAD.
  We will cover data from supported file formats and explore options for
  creating your own schemas and plugins to support new file formats and create custom electronic lab notebooks (ELNs).
  We look at the different ways you can customize your Oasis. Finally, we will show you how to contribute to the development of NOMAD and its ecosystem.

Concept/Content/Structure:

• Title: NOMAD for experimantal materials science data
• Session: Thursday 16th Nov., 13:00 - 16:00h, https://events.fairmat-nfdi.eu/event/2/contributions/65/
• main idea:
  • customizing NOMAD for the needs of experimentalists
  • understanding of schemas in NOMAD
  • from custom schema to standardization and/or community formats
  • demo and hands-on: built-in ELNs, XRD, PLD, MOVPE,
• targeted audience: advanced users, data stewards (people that write schemas and plugins)
• connection to other users meeting sessions: NOMAD user basics in session with Markus --> upload, explore etc. general NOMAD actions, not specific to experimentalists!
• Structure:
  • Intro (only short) (25min talk + 15 discussion) --> 40mins (Sebastian and Sandor)
    • define needs of users regarding ELN/RDM etc.
    • what is structured data and why?
    • what is a schema --> schema in RDM and NOMAD (pic from homepage), analogy picture
    • schema in NOMAD: expressed in yaml, python, main elements: quantitites, attributes, sections, coding a schema (inheritance and Composition)
    • levels of schema in NOMAD
    • standardization part --> nexus
    • different ways of implementing e.g. a new measurement
    • CAMELS?
    • end with discussion or Q&A (15min)
  • Hands-on demo part 1 (40min) (Hampus or Andrea?)
    • demo and hands-on - built-in ELNs (40min):
      • make up story (maybe prepare powder, heat up and measure) --> 10mins
      • familiarize with base section --> user interaction --> 5mins
      • build workflow --> users do it themselfes --> user interaction --> 5mins
      • transfer to custom schema --> demo --> 10mins
      • download yaml add one property --> user interaction --> 10mins
  • Break: ~20mins
  • Hands-on demo part 2
    • XRD (20-30mins) (Rubel and Hampus?)
      • real use case
      • from user schema to standard
      • data reader, community standard --> website, compare structure
      • demo customizing

      • demo Hampus:

        • convert schema to plugin by Hampus program
        • custom plugin
        • community plugin
        • show how plugin works in NOMAD
    • demo of Area A and B examples (30min + 10min questions):
      • PLD (Hampus)
      • MOVPE (Andrea)
      • EBSD (Markus)
      • CAMELS ?

Abstract

This workshop offers insights into customizing and using NOMAD for experimental materials science data. We will explore NOMAD's levels of schemas which are the key to generate structured and well-described research data. We will show how to move from custom user schemas to NOMAD schema plugins and to community standards. Participants will gain hands-on experience with our built-in ELNs. The workshop will showcase use cases we developed within the FAIRmat project, demonstrating the immense potential of NOMAD in advancing your materials science research.

[hampusnasstrom]

Looks good to me! Do we want the users to also follow along on their computers in part 2 as well? If we want that we would probably need to include our plugins in either the central distribution or provide a test OASIS for people to use.

[aalbino2]

For the Hands-on demo part 1: is this supposed to show the base sections available in the global NOMAD, i.e., without any plugin?
Not sure how broad impact this could achieve!

IMO, I would need to show stuff with a yaml file where I use base sections

Let's discuss together what to tell during that session.

[budschi]

built-in ELN

• Story:
  • powder preparation --> composite system
  • annealing/ntering --> process
  • XRD --> measurment --> will be replaced in part 2
  • group everythin in an experiment
  • instruments

[budschi]

extend process by custom yaml schema --> Andrea

[budschi]

story:
powder --> attach XRD file --> nothing happens, fill sth in richeditquantity --> plugin

[budschi]

needs an overview slide to present overall experiment

[RubelMozumder]

In the introductory part, we can also mention Piping data to the Nomad schema from other ontology e.g. Nexus.
In part 1, we can also give an overview of the XRD application definition from NeXus that will give completeness of data piping to the Nomad schema from other ontology.

[budschi]

• Hands-on BUilt.in ELN works
• XRD part should run on central NOMAD --> @Pepe-Marquez asks Markus
• Demo part can run on NOMAD Oasis IKZ

[budschi]

final story of demo/handson:

• powder sintering experiment
• built-in ELN to create: sample, sintering process, XRD measurment, overarching experiment
• replace sintering by custom schema
• replace XRD by custom schema with fileeditquantity and tabularparser
• convert to XRD plugin to read xrdml and do some calculation on it
• Rubel: convert to community standard

• demo part:

• PLD: highlight our plugin and levels of schema again
• MOVPE: highlight different users and multiple plugins, centrality of the ELN for laboratory workflow
• Markus K ?

[aalbino2]

Some notes for the introductory part by Sebastian:
as we were saying, you will introduce some concept functional for the next parts of the worshop, I will list few concepts that I noticed while writing the schema that should be clear to the audience

• Entry
• Section
• Quantity
• composition and inheritance (Object oriented fundamental concepts)

We have now a small custom yaml and a csv file:
Ga2O3_2theta.csv

definitions:
  name: 'A test ELN for user meeting 16.11.23'
  sections:
    Sintering:
      base_sections:
        - nomad.datamodel.metainfo.basesections.Process
        - nomad.datamodel.data.EntryData
      sub_sections:
        steps:
          repeats: True
          section: '#/TemperatureRamp'
    TemperatureRamp:
      m_annotations:
        eln: 
          properties:
            order: 
              - "name"
              - "start_time"
              - "initial_temperature"
              - "final_temperature"
              - "duration"
              - "comment"
      base_sections:
        - nomad.datamodel.metainfo.basesections.ProcessStep
      quantities:
        initial_temperature:
          type: np.float64
          unit: celsius
          description: "initial temperature set for ramp"
          m_annotations:
            eln:
              component: NumberEditQuantity
              defaultDisplayUnit: celsius
        final_temperature:
          type: np.float64
          unit: celsius
          description: "final temperature set for ramp"
          m_annotations:
            eln:
              component: NumberEditQuantity
              defaultDisplayUnit: celsius
    XRDmeasurement:
      m_annotations:
        eln:
      base_sections:
        - nomad.datamodel.metainfo.basesections.Measurement
        - nomad.datamodel.data.EntryData
      sub_sections:
        results:
          repeats: True
          section: '#/XRDmeasurementResults'
    XRDmeasurementResults:
      base_sections:
        - nomad.datamodel.metainfo.basesections.MeasurementResult
        - nomad.datamodel.metainfo.plot.PlotSection
        - nomad.parsing.tabular.TableData
      m_annotations:
        eln:
        plotly_graph_object:
        - data:
            x: '#two_theta'
            y: '#intensity'
          layout:
            title:
              text: '2Theta scan'
          label: '2Theta scan'
          index: 1
      quantities:
        data_file:
          type: str
          description: "Upload here the spreadsheet file containing the XRD data"
          m_annotations:
            tabular_parser:
              parsing_options:
                comment: '#'
              mapping_options:
              - mapping_mode: column
                file_mode: current_entry
                sections:
                - '#root' #results
            browser:
              adaptor: RawFileAdaptor
            eln:
              component: FileEditQuantity
        two_theta:
          type: np.float64
          unit: degree
          shape: ['*']
          description: "2theta"
          m_annotations:
            tabular:
              name: two_theta
        intensity:
          type: np.float64
          shape: ['*']
          description: "Counts per second"
          m_annotations:
            tabular:
              name: CPS

[aalbino2]

I also shared with @budschi and @ka-sarthak an upload in staging NOMAD