solutions/PyONCat.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PyONCat (ONCat API from Python)\n",
    "\n",
    "## Introduction\n",
    "\n",
    "### About\n",
    "\n",
    "ONCat is a metadata catalog built to store information about neutron experiment data at HFIR / SNS.  The contents of the catalog can be viewed at https://oncat.ornl.gov.\n",
    "\n",
    "An API is available to allow programmatic access to the metadata stored in the catalog.  Documentation for the API is at https://oncat.ornl.gov/#/build.\n",
    "\n",
    "This notebook outlines the usage of \"PyONCat\", a Python module built to make communicating with the API a little easier.\n",
    "\n",
    "<p><font color='green'>**(Questions / requests / feedback?  Please contact ONCat Support: oncat-support@ornl.gov.)**</font></p>\n",
    "\n",
    "### Installation\n",
    "\n",
    "The latest version of PyONCat should already be installed on https://jupyter.sns.gov as well as instrument / analysis machines, but if you are using a machine without it then it can be installed using `pip` as follows:\n",
    "\n",
    "```\n",
    "pip install https://oncat.ornl.gov/packages/pyoncat-1.1-py3-none-any.whl\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Usage\n",
    "\n",
    "### 1 - Initial Setup\n",
    "\n",
    "#### Main `ONCat` Object Creation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pyoncat\n",
    "\n",
    "# This is a temporary \"client ID\" intended for use in this tutorial **only**.\n",
    "# For your own work, please contact ONCat Support to be issued your own credentials.\n",
    "CLIENT_ID = \"c0686270-e983-4c71-bd0e-bfa47243a47f\"\n",
    "\n",
    "# Let's use the testing version of ONCat for this tutorial.\n",
    "ONCAT_URL = \"https://oncat-testing.ornl.gov\"\n",
    "\n",
    "oncat = pyoncat.ONCat(\n",
    "    ONCAT_URL,\n",
    "    client_id=CLIENT_ID,\n",
    "    flow=pyoncat.RESOURCE_OWNER_CREDENTIALS_FLOW,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Logging in With the User's Credentials"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import getpass\n",
    "\n",
    "username = getpass.getuser()\n",
    "\n",
    "oncat.login(username, getpass.getpass(\"Enter password for \\\"\" + username + \"\\\":\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2 - Basic Facility / Instrument Information\n",
    "\n",
    "#### Printing the Names of the Facilities Supported by ONCat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "facilities = oncat.Facility.list()\n",
    "\n",
    "for facility in facilities:\n",
    "    print(facility.get(\"name\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Printing the Names of the Instruments Support by ONCat for a Single Facility"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "instruments = oncat.Instrument.list(facility=\"SNS\")\n",
    "\n",
    "for instrument in instruments:\n",
    "    print(instrument.get(\"name\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### EXERCISE A\n",
    "\n",
    "Print out the names of the **HFIR** instruments known to ONCat."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "instruments = oncat.Instrument.list(facility=\"HFIR\")\n",
    "\n",
    "for instrument in instruments:\n",
    "    print(instrument.get(\"name\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3 - Experiment Information\n",
    "\n",
    "#### Retrieving All Experiments for an Instrument"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "experiments = oncat.Experiment.list(facility=\"SNS\", instrument=\"NOM\")\n",
    "\n",
    "for experiment in experiments:\n",
    "    print(experiment.get(\"name\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In general, only calibration experiments and the experiments you have worked on should be visible in the list above.\n",
    "\n",
    "Certain staff members (e.g. NOMAD instrument scientists) will be able to see all the experiments in the list.\n",
    "\n",
    "#### EXERCISE B\n",
    "\n",
    "Print out the names of all the experiments for an instrument you have worked on, either at SNS or HFIR."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "experiments = oncat.Experiment.list(facility=\"SNS\", instrument=\"PG3\")\n",
    "\n",
    "for experiment in experiments:\n",
    "    print(experiment.get(\"name\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Getting All the Information We Have for a given Experiment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's use a calibration experiment that everyone has access to.\n",
    "nom_cal_exp = oncat.Experiment.retrieve(\n",
    "    \"IPTS-19564\",\n",
    "    facility=\"SNS\",\n",
    "    instrument=\"NOM\"\n",
    ")\n",
    "\n",
    "nom_cal_exp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the object we got back was an `ONCatRepresentation`.  This is just a slightly more convenient wrapper around the information we got back from the API, which has a nested, \"dictionary of dictionaries\" structure.\n",
    "\n",
    "#### Accessing Fields Using `.get()`\n",
    "\n",
    "Similarly to when we printed out the names of facilities and instruments, we can directly access the various bits of information above using `.get()`, for example when printing the title:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nom_cal_exp.get(\"title\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This syntax allows you to \"drill down\" into the deeply-nested fields within the structure using dot-delimited paths.  For example, there is a string containing a comma-separated list of all the runs that were taken during the experiment.  This is at `indexed` -> `run_number` -> `ranges` in the structure above, and we can directly access that information as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nom_cal_exp.get(\"indexed.run_number.ranges\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### EXERCISE C\n",
    "\n",
    "As a quick way to get an idea of when an experiment took place, we store the the `earliest` and `latest` time at which a datafile has been `created`.\n",
    "\n",
    "Look again at all the experiment information, and use `.get()` to print out both of those times on the `nom_cal_exp` object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(nom_cal_exp.get(\"earliest.created\"))\n",
    "print(nom_cal_exp.get(\"latest.created\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4 - Datafile Information\n",
    "\n",
    "#### Retrieving All Datafiles for an Experiment\n",
    "\n",
    "Let's get all the datafiles for the same calibration experiment we looked at previously:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "datafiles = oncat.Datafile.list(\n",
    "    facility=\"SNS\",\n",
    "    instrument=\"NOM\",\n",
    "    experiment=\"IPTS-19564\",\n",
    ")\n",
    "\n",
    "len(datafiles)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at what a single entry for a datafile contains:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "datafiles[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, quite a lot of stuff...  This is because we ingest as much of the metadata as we can from the datafiles as they come off the beamlines.\n",
    "\n",
    "#### Accessing Information on Datafiles \n",
    "\n",
    "The datafile objects we get from the API can be accessed in much the same way as the experiment objects we looked at before, except different information is stored.\n",
    "\n",
    "Every datafile has a location:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "datafiles[0].get(\"location\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "SNS instruments (and WAND<sup>2</sup>) work in terms of \"runs\", so datafiles will have a corresponding run number:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "datafiles[0].get(\"indexed.run_number\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We store when datafiles are created:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "datafiles[0].get(\"created\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But the vast majority of the remaining info is nested inside the metadata field:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "datafiles[0].get(\"metadata\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Easily Seeing All Fields at a Glace\n",
    "\n",
    "With all that metadata it can be hard to find what you're looking for.\n",
    "\n",
    "Luckily, there is an easier way to see all the dot-delimited paths in a given datafile:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "datafiles[0].nodes()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### EXERCISE D\n",
    "\n",
    "Find which dot-limited path corresponds to the average rotation speed of chopper 1, and then use that path to print out the speed for all the runs in the experiment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for datafile in datafiles:\n",
    "    print(datafile.get(\"metadata.entry.instrument.chopper1.rotation_speed.average_value\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Filtering by Fields Using \"Projections\"\n",
    "\n",
    "It is also possible to ask for a much smaller sub-set of information for each datafile, using something called a projection.  This means that we can ask for exactly what we need for a very large number of datafiles and not have to wait too long for a result.\n",
    "\n",
    "A projection is just a list of strings, where each item is the same kind of dot-delimeted paths we were working with previously.  For example, a projection that retrieved run number and sample name for each datafile might look like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "projection = [\n",
    "    \"indexed.run_number\",\n",
    "    \"metadata.entry.sample.name\",\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We could then use that projection to get run number and sample name information back for *all* the experiments we have access to:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"This may takea a little while...  Please wait.\")\n",
    "\n",
    "datafiles = oncat.Datafile.list(\n",
    "    facility=\"SNS\",\n",
    "    instrument=\"NOM\",\n",
    "    projection=projection,\n",
    ")\n",
    "\n",
    "print(\"Done!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each datafile will now contain far less information:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "datafiles[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### FINAL EXERCISE - Putting it All Together\n",
    "\n",
    "Now is the time to take everything you have learned and to put it together to carry out a search for a particular set of runs.\n",
    "\n",
    "Searching across **all** runs available to you on NOMAD, print the **proton charge**, **total counts** and **location** of the runs that meet **all three** of the following criteria:\n",
    "\n",
    "1. a sample name equal to `\"V rod\"`,\n",
    "\n",
    "2. a proton charge of between `1.9e12` and `2.1e12`; and\n",
    "\n",
    "3. a total count greater than `1.0e9`.\n",
    "\n",
    "#### Step 1\n",
    "\n",
    "Get a list of all datafiles which includes all the information we'd like to search through:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "projection = [\n",
    "    \"location\",\n",
    "    \"metadata.entry.sample.name\",\n",
    "    \"metadata.entry.proton_charge\",\n",
    "    \"metadata.entry.total_counts\",\n",
    "]\n",
    "\n",
    "datafiles = oncat.Datafile.list(\n",
    "    facility=\"SNS\",\n",
    "    instrument=\"NOM\",\n",
    "    projection=projection,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 2\n",
    "\n",
    "Loop over each datafile in the list and only print the required information for the runs we need:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for datafile in datafiles:\n",
    "    if datafile.get(\"metadata.entry.sample.name\") != \"V rod\":\n",
    "        continue\n",
    "    if not (1.9e12 < datafile.get(\"metadata.entry.proton_charge\") < 2.1e12):\n",
    "        continue\n",
    "    if datafile.get(\"metadata.entry.total_counts\") < 1.0e9:\n",
    "        continue\n",
    "    print(\n",
    "        datafile.get(\"metadata.entry.proton_charge\"),\n",
    "        datafile.get(\"metadata.entry.total_counts\"),\n",
    "        datafile.get(\"location\"),\n",
    "    )"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "python3-default at jnrk-sns-analysis",
   "language": "python",
   "name": "jnrk-sns-analysis-python3-default"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}