pyhep2022_choi.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Management Package for the novel data delivery system, ServiceX, and its application to an ATLAS Run-2 Physics Analysis Workflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### KyungEon Choi (UT Austin) - PyHEP 2022 - Sep 14, 2022"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/logo_servicex.png\" width=\"100\" height=\"100\"  style=\"float:left\" alt=\"ServiceX\">\n",
    "\n",
    "# ServiceX"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=\"3\">\n",
    "    A service to <b><span style=\"color:red;\">quickly</span></b> access a \"fraction\" of large data on the grid \n",
    "\n",
    "<img src=\"img/ServiceX_diagram_1.pdf\" height=\"140\"  style=\"float:center\" alt=\"ServiceX-1\">\n",
    "\n",
    "\n",
    "If an analyzer wants to see a distribution of a variable (or column or branch) of large data on the grid, one needs to submit a grid job or download all files.\n",
    "\n",
    "<br>\n",
    "<br>\n",
    "<br>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=\"3\">\n",
    "    ServiceX is deployed in a Kubernetes cluster* and usually co-located with the data center to allow a wide network bandwidth.\n",
    "</font>\n",
    "\n",
    "<font size=\"2\" style=\"float:right\">*Container orchestration system</font>\n",
    "\n",
    "<br>\n",
    "\n",
    "<img src=\"img/ServiceX_diagram_2.pdf\" height=\"140\"  style=\"float:center\" alt=\"ServiceX-1\">\n",
    "\n",
    "<br>\n",
    "\n",
    "<font size=\"3\">\n",
    "    It spawns the so-called <b><span style=\"color:blue\">Transformer</span></b> pods to read columns and filter events from each file in parallel. Transformer pods are also scaled automatically using Kubernetes auto-scaling feature. Outputs from each transformer are written to object store and then delivered to a destination as soon as it becomes available.\n",
    "</font>\n",
    "<img src=\"img/ServiceX_diagram_3.pdf\" height=\"140\"  style=\"float:center\" alt=\"ServiceX-1\">\n",
    "\n",
    "<br>\n",
    "\n",
    "<font size=\"3\">\n",
    "    Transformers are separated by the input data format. Currently transformers for two types of data format are maintained: \n",
    "    <ul style=\"line-height:1.6;\">\n",
    "        <li><b>Uproot transformer for flat ROOT ntuple (e.g. CMS nanoAOD)</b></li>\n",
    "        <li><b>xAOD transformer for ATLAS xAOD data</b></li>\n",
    "    </ul>\n",
    "    Other types of transformers are also under development, including transformer for non-LHC HEP experiment.\n",
    "</font>\n",
    "\n",
    "<br>\n",
    "<br>\n",
    "<br>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=\"3\">\n",
    "    <b><span style=\"color:blue\">FuncADL</span></b> (Functional Analysis Description Language) is the query language that the two transformers understand to filter events and select columns. Details on the grammar and capabilities of FuncADL can be found from Mason's talk at PyHEP 2021.\n",
    "\n",
    "<a href=\"https://indico.cern.ch/event/1019958/timetable/#31-funcadl-functional-analysis\" target=\"_blank\">\n",
    "    <img src=\"img/funcadl.png\" height=\"140\"  style=\"float:center\" alt=\"ServiceX-1\">\n",
    "</a>\n",
    "\n",
    "The basic operators of FuncADL will be shown in this notebook: <code>Select()</code> to select columns and <code>Where()</code> to apply cuts.\n",
    "\n",
    "<b><span style=\"color:blue\">ROOT TCut syntax</span></b> is also supported to provide a more familiar experience:\n",
    "- Supported expressions\n",
    "  - Arithmetic operators: `+, -, *, /`\n",
    "  - Logical operators: `!, &&, ||`\n",
    "  - Relational and comparison operators: `==, !=, >, <, >=, <=`\n",
    "  - Ternary operator: `(A?B:C)` - has to be enclosed in parentheses\n",
    "  - Mathematical function: `sqrt`\n",
    "- Limitations\n",
    "    - Compatible with input data format of ROOT flat ntuple\n",
    "    - Cuts can be applied to scalar type branches (crashes if a datatype of TTree branch is `std::vector`)\n",
    "\n",
    "\n",
    "<br>\n",
    "<br>\n",
    "<br>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=\"3\">\n",
    "\n",
    "The following <b><span style=\"color:blue\">ServiceX endpoints</span></b> are  currently available for users to make data delivery requests.\n",
    "\n",
    "| Type | Collaboration | Input data format | Location | Endpoint |\n",
    "| :----: |:----: | :-----------------: | :--------: | :--------: |\n",
    "| stand-alone | ATLAS | ROOT flat ntuple | UC Analysis Facility | https://uproot-atlas.servicex.af.uchicago.edu/ |\n",
    "| stand-alone | ATLAS |xAOD | UC Analysis Facility | https://xaod.servicex.af.uchicago.edu/ |\n",
    "| coffea-casa | CMS | ROOT flat ntuple | UNL | https://coffea.casa/ |\n",
    "| coffea-casa | - | ROOT flat ntuple | UNL | https://coffea-opendata.casa |\n",
    "\n",
    "- Type\n",
    "    - `stand-alone`: Secured by own authentication system. An endpoint is accessible from anywhere with an access token.\n",
    "    - `coffea-casa`: Integrated into a coffea-casa facility for CMS. Secured by CERN authentication system of coffea-casa facility and an ServiceX endpoint is not exposed to outside.\n",
    "    \n",
    "- Collaboration\n",
    "    - Access to the experimental data\n",
    "- Input data format\n",
    "    - Dedicate endpoint for different types of input data format (or transformer)\n",
    "- Location\n",
    "    - ATLAS endpoints at University of Chicago Analysis Facility (UC AF) and CMS endpoints at University of Nebraska-Lincoln (UNL) are activly maintained.\n",
    "\n",
    "\n",
    "<br>\n",
    "<br>\n",
    "<br>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=\"3\">\n",
    "\n",
    "<b><span style=\"color:blue\">Access to a ServiceX endpoint</span></b> requires a yaml file containing access information.\n",
    "\n",
    "- A yaml file can have multiple endpoints\n",
    "- Optionally other information (such as `cache_path`) can be placed\n",
    "- For stand-alone endpoints, a config file can be downloaded from the webpage\n",
    "- The config file can be called `servicex.yaml`, `servicex.yml`, or `.servicex`. The files are searched in that order, and all present are used.\n",
    "- Searches in the current working directory, and above, and home directory (`$HOME` on Linux and Mac, and profile directory on Windows)\n",
    "\n",
    "```\n",
    "api_endpoints:\n",
    "  - name: <your-endpoint-name>\n",
    "    endpoint: <your-endpoint>\n",
    "    token: <api-token>\n",
    "    type: uproot\n",
    "```\n",
    "\n",
    "<br>\n",
    "<br>\n",
    "<br>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=\"3\">\n",
    "    \n",
    "<b><span style=\"color:blue\">Local data cache</span></b> \n",
    "\n",
    "- ServiceX queries and data are stored in a local temporary directory by default\n",
    "- The same query on the same dataset pulls the data from the local cache\n",
    "- The cache is unbound: it will continuously fill up\n",
    "- Cache path can (or need to) be changed in the configuration file\n",
    "\n",
    "```\n",
    "cache_path: <cache-path>\n",
    "api_endpoints:\n",
    "  - name: <your-endpoint-name>\n",
    "    endpoint: <your-endpoint>\n",
    "    token: <api-token>\n",
    "    type: uproot\n",
    "```\n",
    "\n",
    "<br>\n",
    "<br>\n",
    "<br>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ServiceX DataBinder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<font size=\"3\">\n",
    "    \n",
    "<b><span style=\"color:blue\">ServiceX DataBinder</span></b> is a Python library for easy manipulations of ServiceX delivery requests and delivered data using a single configuration file.\n",
    "\n",
    "<b>All you need to prepare are `servicex.yaml` and `servicex-databinder` to make use of ServiceX.</b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# %%writefile servicex.yaml\n",
    "# api_endpoints:\n",
    "#   - name: opendata_uproot_river\n",
    "#     endpoint: https://atlasopendata.servicex.ssl-hep.org/\n",
    "#     type: uproot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install servicex-databinder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import servicex_databinder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "servicex_databinder.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from servicex_databinder import DataBinder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<font size=\"3\">\n",
    "\n",
    "Here is the workflow with ServiceX DataBinder once you have your `servicex.yaml` file in place and installed `servicex-databinder`:\n",
    "\n",
    "<img src=\"img/ServiceX_workflow_PyHEP.pdf\" width=900 />\n",
    "\n",
    "1. A user write a configuration for ServiceX DataBinder\n",
    "1. Make ServiceX delivery requests from Jupyter notebook\n",
    "1. ServiceX backend looks for input datasets and retrieves an input file list\n",
    "1. A relevant code is generated based on the input data format, query in func-adl, and so on\n",
    "1. Transformer pods (workers) are generated to process each file (10 pods at first and scale up if necessary)\n",
    "1. Outputs are written to object store inside the Kubernetes cluster\n",
    "1. Download outputs to the local cache directory asynchronously\n",
    "1. Copy files to the final destination\n",
    "\n",
    "Step 1 and 2 are done by a user.\n",
    "\n",
    "<br>\n",
    "<br>\n",
    "<br>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=\"4\">\n",
    "    \n",
    "<b><span style=\"color:blue\"> Use-case 1. </span></b> An analyzer simply wants to see the electron pT distribution of large dataset on the grid\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=\"3\">   \n",
    "Let's write a configuration file. \n",
    "    \n",
    "In `General` block:\n",
    "- `ServiceXBackendName` is the endpoint name in your ServiceX access file (e.g. `servicex.yaml`)\n",
    "- `OutputDirectory` is the base output directory\n",
    "- `OutputFormat` is the format of ServiceX delivered files. Either `parquet` or `root` supported for Uproot\n",
    "- `WriteOutputDict` is the name of an ouput yaml file containing a dictionary of output file paths\n",
    "    \n",
    "In `Sample` block follows:\n",
    "- `Name` is the name of sample.\n",
    "- `XRootDFiles` is the input files. These 50 files are CMS OpenData converted to flat ROOT ntuple and amount to about 10GB.\n",
    "- `Tree` is the name of ROOT TTree that I want to read.\n",
    "- `FuncADL` is the query that I want to pass to the transformers.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%%writefile config_example_1.yml\n",
    "General:\n",
    "  ServiceXBackendName: uproot_uc_af\n",
    "  OutputDirectory: ServiceXData_1\n",
    "  OutputFormat: parquet\n",
    "  WriteOutputDict: out_example_1\n",
    "\n",
    "Sample:\n",
    "  - Name: ttbar\n",
    "    XRootDFiles: https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/00DF0A73-17C2-E511-B086-E41D2D08DE30.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/020D0AF1-4BC2-E511-BDFC-0026B95ADB18.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/02837459-03C2-E511-8EA2-002590A887AC.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/0400D1B1-4CC2-E511-AD14-AC853DA06A1A.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/0634456A-08C2-E511-A0C1-001E6739722E.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/0A5A5364-03C2-E511-9352-00259048AE00.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/0A5C7429-0CC2-E511-BC6B-002590D9D8B8.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/0AE230FC-4BC2-E511-B35C-000F530E4644.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/0C5BB11A-E2C1-E511-8C02-002590A831B6.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/10D0AF2E-33C2-E511-BCE5-001E67398683.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/14C8A87C-DCC1-E511-BE67-0CC47A0AD704.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/14F4047E-0CC2-E511-B1CE-002590A3C970.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/16467C82-D8C1-E511-9546-001E67E71417.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/167A53AB-08C2-E511-A58B-001E67397F2B.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/18897870-17C2-E511-9DC5-E41D2D08DCA0.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/18CCDFE4-02C2-E511-B0FA-002590A37114.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/20A0646D-D8C1-E511-9587-001E67396A09.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/2235F83F-DBC1-E511-A83F-001E673986B5.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/22761DF1-E1C1-E511-8F86-003048CB8584.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/244D8186-D8C1-E511-B684-001E67E71BF5.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/26CAB9BA-CEC1-E511-8B0B-001E67398E49.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/2A18493A-04C2-E511-935C-002590D9D8C2.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/2C7F8762-0CC2-E511-9F77-001E673981C4.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/2CA9FB72-17C2-E511-A3E3-0025907DCA7E.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/2CCB8F4A-07C2-E511-A3C6-002590A3C970.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/2EF6150C-1AC2-E511-A67B-002590A83192.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/324D4BDC-DCC1-E511-925D-00259029E922.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/32D38D09-E2C1-E511-986A-001E67397BC5.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/32E942A0-4DC2-E511-AD4A-001E674FCBF7.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/343CDCDC-E1C1-E511-80C4-00259029E91C.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/36171880-02C2-E511-9B6E-0CC47A0AD630.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/363584E8-4BC2-E511-9DE3-008CFAF060D8.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/3660CB69-35C2-E511-86B8-002590A83354.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/366A2A31-D8C1-E511-8A68-001E67396E32.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/3C73C475-08C2-E511-B273-001E67E71943.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/3E146760-D8C1-E511-A6E1-0CC47A57D136.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/401D0082-08C2-E511-9934-001E6739677A.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/4207E269-0CC2-E511-BB42-001E6739801B.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/42110E2D-13C2-E511-9460-0025B3E064F8.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/46D80108-06C2-E511-B908-001E67E7136D.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/48B429A9-D2C1-E511-BED7-002590A80E1C.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/4C675E0C-01C2-E511-B25E-002590D9D96E.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/4EF74313-E2C1-E511-88F6-001E67397396.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/503646F9-35C2-E511-8CF5-001E6739751C.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/52CAF991-DBC1-E511-B168-00259048A860.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/547D8379-D8C1-E511-9EEA-00259048A8F0.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/563553BB-D5C1-E511-826F-001E67E6F89B.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/56535980-D8C1-E511-9C47-001E67E71DDA.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/58126B26-D8C1-E511-8BDA-002481E14E48.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/5879C6E0-00C2-E511-958C-0CC47A0AD630.root\n",
    "    Tree: events    \n",
    "    FuncADL: \"Where(lambda event: event.nelectron_pt > 5). \\\n",
    "        Select(lambda event: {'electron_pt': event.electron_pt})\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=\"3\">\n",
    "    \n",
    "Load the configuration file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sx = DataBinder(\"config_example_1.yml\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=\"3\">\n",
    "    \n",
    "Make a ServiceX delivery request!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "out = sx.deliver()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=\"3\">\n",
    "\n",
    "In addition to the progress bar shown above, you can monitor status of transformations in the dashboard (only for stand-alone endpoints):\n",
    "[UC AF Uproot Dashboard](https://uproot-atlas.servicex.af.uchicago.edu/dashboard) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=\"3\">\n",
    "\n",
    "The returned object by `deliver()` is a dictionary of delivered files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"Samples: {out.keys()}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"Trees in sample ttbar: {out['ttbar'].keys()}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"Number of delivery files under ttbar/events: {len(out['ttbar']['events'])}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=\"3\">\n",
    "    \n",
    "Let's make a plot of what we got."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import awkward as ak\n",
    "import hist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ak_el_pt = ak.from_parquet(out['ttbar']['events'])['electron_pt']\n",
    "print(f\"Delivered electron pT column: {ak_el_pt}\")\n",
    "print(f\"Number of events: {len(ak_el_pt)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "h = hist.Hist(hist.axis.Regular(50, 0, 100, name=\"Electron pT\"))\n",
    "h.fill(ak.flatten(ak_el_pt))\n",
    "h.plot();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br>\n",
    "<br>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=\"3\">\n",
    "    \n",
    "FuncADL expression can be replaced by ROOT TCut syntax:<br>\n",
    "\n",
    "```\n",
    "FuncADL: \"Where(lambda event: event.nelectron_pt > 5). \\\n",
    "        Select(lambda event: {'electron_pt': event.electron_pt})\"\n",
    "```\n",
    "    to\n",
    "\n",
    "```\n",
    "Filter: nelectron_pt > 5\n",
    "Columns: electron_pt\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile config_example_2.yml\n",
    "General:\n",
    "  ServiceXBackendName: uproot_uc_af\n",
    "  OutputDirectory: ServiceXData_2\n",
    "  OutputFormat: parquet\n",
    "  WriteOutputDict: out_example_2\n",
    "\n",
    "Sample:\n",
    "  - Name: ttbar\n",
    "    XRootDFiles: https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/00DF0A73-17C2-E511-B086-E41D2D08DE30.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/020D0AF1-4BC2-E511-BDFC-0026B95ADB18.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/02837459-03C2-E511-8EA2-002590A887AC.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/0400D1B1-4CC2-E511-AD14-AC853DA06A1A.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/0634456A-08C2-E511-A0C1-001E6739722E.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/0A5A5364-03C2-E511-9352-00259048AE00.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/0A5C7429-0CC2-E511-BC6B-002590D9D8B8.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/0AE230FC-4BC2-E511-B35C-000F530E4644.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/0C5BB11A-E2C1-E511-8C02-002590A831B6.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/10D0AF2E-33C2-E511-BCE5-001E67398683.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/14C8A87C-DCC1-E511-BE67-0CC47A0AD704.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/14F4047E-0CC2-E511-B1CE-002590A3C970.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/16467C82-D8C1-E511-9546-001E67E71417.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/167A53AB-08C2-E511-A58B-001E67397F2B.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/18897870-17C2-E511-9DC5-E41D2D08DCA0.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/18CCDFE4-02C2-E511-B0FA-002590A37114.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/20A0646D-D8C1-E511-9587-001E67396A09.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/2235F83F-DBC1-E511-A83F-001E673986B5.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/22761DF1-E1C1-E511-8F86-003048CB8584.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/244D8186-D8C1-E511-B684-001E67E71BF5.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/26CAB9BA-CEC1-E511-8B0B-001E67398E49.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/2A18493A-04C2-E511-935C-002590D9D8C2.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/2C7F8762-0CC2-E511-9F77-001E673981C4.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/2CA9FB72-17C2-E511-A3E3-0025907DCA7E.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/2CCB8F4A-07C2-E511-A3C6-002590A3C970.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/2EF6150C-1AC2-E511-A67B-002590A83192.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/324D4BDC-DCC1-E511-925D-00259029E922.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/32D38D09-E2C1-E511-986A-001E67397BC5.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/32E942A0-4DC2-E511-AD4A-001E674FCBF7.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/343CDCDC-E1C1-E511-80C4-00259029E91C.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/36171880-02C2-E511-9B6E-0CC47A0AD630.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/363584E8-4BC2-E511-9DE3-008CFAF060D8.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/3660CB69-35C2-E511-86B8-002590A83354.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/366A2A31-D8C1-E511-8A68-001E67396E32.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/3C73C475-08C2-E511-B273-001E67E71943.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/3E146760-D8C1-E511-A6E1-0CC47A57D136.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/401D0082-08C2-E511-9934-001E6739677A.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/4207E269-0CC2-E511-BB42-001E6739801B.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/42110E2D-13C2-E511-9460-0025B3E064F8.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/46D80108-06C2-E511-B908-001E67E7136D.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/48B429A9-D2C1-E511-BED7-002590A80E1C.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/4C675E0C-01C2-E511-B25E-002590D9D96E.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/4EF74313-E2C1-E511-88F6-001E67397396.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/503646F9-35C2-E511-8CF5-001E6739751C.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/52CAF991-DBC1-E511-B168-00259048A860.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/547D8379-D8C1-E511-9EEA-00259048A8F0.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/563553BB-D5C1-E511-826F-001E67E6F89B.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/56535980-D8C1-E511-9C47-001E67E71DDA.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/58126B26-D8C1-E511-8BDA-002481E14E48.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/5879C6E0-00C2-E511-958C-0CC47A0AD630.root\n",
    "    Tree: events    \n",
    "    Filter: nelectron_pt > 5\n",
    "    Columns: electron_pt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sx = DataBinder(\"config_example_2.yml\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "out_2 = sx.deliver()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=\"3\">\n",
    "\n",
    "Since the same query on the same dataset is found at the local cache, the delivery request is not propagated to the ServiceX and data is pulled from the local cache. \n",
    "\n",
    "A change in the query and/or name of dataset will initiate new ServiceX data delivery.\n",
    "\n",
    "<br>\n",
    "<br>\n",
    "<br>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=\"3\">\n",
    "\n",
    "It's more practical to use experimental data behind an auth, thus `RucioDID` will be typical  type of dataset (supported by ServiceX).\n",
    "\n",
    "Let's run on a rucio dataset, which amounts to 11 GB (114 files) at BNL and Munich ATLAS storage (Only ATLAS member can get an access token for ATLAS ServiceX endpoints)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile config_example_3.yml\n",
    "General:\n",
    "  ServiceXBackendName: uproot_river_internal\n",
    "  OutputDirectory: ServiceXData_3\n",
    "  OutputFormat: parquet\n",
    "  WriteOutputDict: out_example_3\n",
    "\n",
    "Sample:\n",
    "  - Name: singleTop\n",
    "    RucioDID: user.ivukotic:user.ivukotic.single_top_s_chan__nominal\n",
    "    Tree: events    \n",
    "    Filter: nelectron_pt > 5\n",
    "    Columns: electron_pt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "out_3 = DataBinder(\"config_example_3.yml\").deliver()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=\"4\">\n",
    "    \n",
    "<b><span style=\"color:blue\"> Use-case 2. </span></b> \n",
    "An analyzer wants to exploit ServiceX for physics analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=\"3\">\n",
    "\n",
    "There are diverse ways to get results from large datasets on data centers. Every analysis has own problems and even various needs within an analysis team. The following schematic shows three example analysis workflows with ServiceX:\n",
    "- <span style=\"color:blue\">Delivery of ROOT ntuple (dashed)</span>: minimal change with respect to the traditional workflow. An analyzer gets selected and filtered branches in ROOT format via ServiceX, and performs subsequent analyses with familiar tools.\n",
    "<br>\n",
    "\n",
    "- <span style=\"color:blue\">ServiceX + Coffea (thick)</span>: more toward Python ecosystem. ServiceX delivers Apache parquet files containing awkward arrays, and a core analysis is done using coffea framework.\n",
    "<br>\n",
    "\n",
    "- <span style=\"color:blue\">ServiceX + Coffea + Cabinetry (pyhf) (thick + thin)</span>: analysis fully in Python ecosystem. \n",
    "\n",
    "\n",
    "<br>\n",
    "\n",
    "<img src=\"img/workflows\" width=900 />\n",
    "\n",
    "<br>\n",
    "<br>\n",
    "<br>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=\"3\">\n",
    "\n",
    "<b><span style=\"color:blue\">Delivery of ROOT files</span></b> can be done by setting \n",
    "\n",
    "    General:\n",
    "      OutputFormat: root\n",
    "\n",
    "in a configuration file and \n",
    "the structure of input ROOT ntuple is mostly preserved in the delivered ROOT files."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=\"3\">\n",
    "    \n",
    "The following configuration file contains three samples (`data`, `ttbar`, `wjets`) with the same query and the output format is set to `root`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile config_example_4.yml\n",
    "General:\n",
    "  ServiceXBackendName: uproot_uc_af\n",
    "  OutputDirectory: ServiceXData_4\n",
    "  OutputFormat: root\n",
    "  WriteOutputDict: out_example_4\n",
    "  IgnoreServiceXCache: True\n",
    "Sample:\n",
    "  - Name: ttbar\n",
    "    XRootDFiles: https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/00DF0A73-17C2-E511-B086-E41D2D08DE30.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/020D0AF1-4BC2-E511-BDFC-0026B95ADB18.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/02837459-03C2-E511-8EA2-002590A887AC.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/0400D1B1-4CC2-E511-AD14-AC853DA06A1A.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/0634456A-08C2-E511-A0C1-001E6739722E.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/0A5A5364-03C2-E511-9352-00259048AE00.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/0A5C7429-0CC2-E511-BC6B-002590D9D8B8.root                \n",
    "    Tree: events    \n",
    "    FuncADL: \"Where(lambda event: event.nelectron_pt > 5). \\\n",
    "            Select(lambda event: {'electron_pt': event.electron_pt})\"\n",
    "  - Name: wjets\n",
    "    XRootDFiles: https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/WJetsToLNu_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/MINIAODSIM/PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext2-v1/10000/002B75B9-10DB-E511-915B-02163E017888.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/WJetsToLNu_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/MINIAODSIM/PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext2-v1/10000/006E9C0C-12DB-E511-981C-00261894397D.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/WJetsToLNu_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/MINIAODSIM/PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext2-v1/10000/0097C77B-12DB-E511-AAC0-02163E011796.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/WJetsToLNu_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/MINIAODSIM/PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext2-v1/10000/00B697CF-10DB-E511-A9D9-782BCB407CFD.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/WJetsToLNu_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/MINIAODSIM/PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext2-v1/10000/00BC9D60-0FDB-E511-A038-00215AD4D670.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/WJetsToLNu_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/MINIAODSIM/PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext2-v1/10000/00D29FE6-0CDB-E511-8602-BCAEC53F6D4E.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/WJetsToLNu_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/MINIAODSIM/PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext2-v1/10000/00F4DF03-10DB-E511-B5D3-842B2B185C54.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/WJetsToLNu_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/MINIAODSIM/PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext2-v1/10000/0228909F-10DB-E511-830F-002481E0DA4E.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/WJetsToLNu_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/MINIAODSIM/PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext2-v1/10000/027FF218-10DB-E511-8FAC-02163E013669.root\n",
    "    Tree: events\n",
    "    FuncADL: \"Where(lambda event: event.nelectron_pt > 5). \\\n",
    "            Select(lambda event: {'electron_pt': event.electron_pt})\"\n",
    "  - Name: data\n",
    "    XRootDFiles: https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/Run2015D/SingleMuon/MINIAOD/16Dec2015-v1/10000/00006301-CAA8-E511-AD39-549F35AD8BC9.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/Run2015D/SingleMuon/MINIAOD/16Dec2015-v1/10000/0034202D-A3A8-E511-BA9C-00259073E3DA.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/Run2015D/SingleMuon/MINIAOD/16Dec2015-v1/10000/0043758E-ECA8-E511-B849-002618FDA287.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/Run2015D/SingleMuon/MINIAOD/16Dec2015-v1/10000/004C08BC-C8A8-E511-943C-00266CFAE6E0.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/Run2015D/SingleMuon/MINIAOD/16Dec2015-v1/10000/005416D9-E0A8-E511-8AA1-0CC47A4C8E46.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/Run2015D/SingleMuon/MINIAOD/16Dec2015-v1/10000/00649463-BEA8-E511-9266-0CC47A4D769C.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/Run2015D/SingleMuon/MINIAOD/16Dec2015-v1/10000/00AB7295-C6A8-E511-AEAF-0CC47A4C8E8A.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/Run2015D/SingleMuon/MINIAOD/16Dec2015-v1/10000/00E42A7E-BAA8-E511-B23B-0CC47A4C8E86.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/Run2015D/SingleMuon/MINIAOD/16Dec2015-v1/10000/00F3E56E-BEA8-E511-A6A2-3417EBE722FA.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/Run2015D/SingleMuon/MINIAOD/16Dec2015-v1/10000/02004059-D9A8-E511-8790-0CC47A78A440.root\n",
    "    Tree: events\n",
    "    FuncADL: \"Where(lambda event: event.nelectron_pt > 5). \\\n",
    "            Select(lambda event: {'electron_pt': event.electron_pt})\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "out_4 = DataBinder(\"config_example_4.yml\").deliver()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=\"3\">\n",
    "    \n",
    "Separate ServiceX requests for each sample and all run in parallal (if possible).\n",
    "\n",
    "The returned file list has keys for all samples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "out_4.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=\"3\">\n",
    "\n",
    "Let's have a look a ROOT file from `ttbar` sample using `uproot`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import uproot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with uproot.open(out_4['ttbar'][0]) as root_file:\n",
    "    print(root_file.keys())\n",
    "    tree = root_file['events']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tree.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br>\n",
    "<br>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=\"3\">\n",
    "\n",
    "The only difference in ServiceX DataBinder for <b><span style=\"color:blue\">ServiceX + Coffea</span></b> workflow is setting\n",
    "\n",
    "    General:\n",
    "      OutputFormat: parquet\n",
    "\n",
    "in a configuration file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile config_example_5.yml\n",
    "General:\n",
    "  ServiceXBackendName: uproot_uc_af\n",
    "  OutputDirectory: ServiceXData_5\n",
    "  OutputFormat: parquet\n",
    "  WriteOutputDict: out_example_5\n",
    "Sample:\n",
    "  - Name: ttbar\n",
    "    XRootDFiles: https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/00DF0A73-17C2-E511-B086-E41D2D08DE30.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/020D0AF1-4BC2-E511-BDFC-0026B95ADB18.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/02837459-03C2-E511-8EA2-002590A887AC.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/0400D1B1-4CC2-E511-AD14-AC853DA06A1A.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/0634456A-08C2-E511-A0C1-001E6739722E.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/0A5A5364-03C2-E511-9352-00259048AE00.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM//PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/00000/0A5C7429-0CC2-E511-BC6B-002590D9D8B8.root                \n",
    "    Tree: events    \n",
    "    FuncADL: \"Where(lambda event: event.nelectron_pt > 5). \\\n",
    "            Select(lambda event: {'electron_pt': event.electron_pt})\"\n",
    "  - Name: wjets\n",
    "    XRootDFiles: https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/WJetsToLNu_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/MINIAODSIM/PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext2-v1/10000/002B75B9-10DB-E511-915B-02163E017888.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/WJetsToLNu_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/MINIAODSIM/PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext2-v1/10000/006E9C0C-12DB-E511-981C-00261894397D.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/WJetsToLNu_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/MINIAODSIM/PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext2-v1/10000/0097C77B-12DB-E511-AAC0-02163E011796.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/WJetsToLNu_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/MINIAODSIM/PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext2-v1/10000/00B697CF-10DB-E511-A9D9-782BCB407CFD.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/WJetsToLNu_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/MINIAODSIM/PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext2-v1/10000/00BC9D60-0FDB-E511-A038-00215AD4D670.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/WJetsToLNu_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/MINIAODSIM/PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext2-v1/10000/00D29FE6-0CDB-E511-8602-BCAEC53F6D4E.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/WJetsToLNu_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/MINIAODSIM/PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext2-v1/10000/00F4DF03-10DB-E511-B5D3-842B2B185C54.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/WJetsToLNu_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/MINIAODSIM/PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext2-v1/10000/0228909F-10DB-E511-830F-002481E0DA4E.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/RunIIFall15MiniAODv2/WJetsToLNu_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/MINIAODSIM/PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext2-v1/10000/027FF218-10DB-E511-8FAC-02163E013669.root\n",
    "    Tree: events\n",
    "    FuncADL: \"Where(lambda event: event.nelectron_pt > 5). \\\n",
    "            Select(lambda event: {'electron_pt': event.electron_pt})\"\n",
    "  - Name: data\n",
    "    XRootDFiles: https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/Run2015D/SingleMuon/MINIAOD/16Dec2015-v1/10000/00006301-CAA8-E511-AD39-549F35AD8BC9.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/Run2015D/SingleMuon/MINIAOD/16Dec2015-v1/10000/0034202D-A3A8-E511-BA9C-00259073E3DA.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/Run2015D/SingleMuon/MINIAOD/16Dec2015-v1/10000/0043758E-ECA8-E511-B849-002618FDA287.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/Run2015D/SingleMuon/MINIAOD/16Dec2015-v1/10000/004C08BC-C8A8-E511-943C-00266CFAE6E0.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/Run2015D/SingleMuon/MINIAOD/16Dec2015-v1/10000/005416D9-E0A8-E511-8AA1-0CC47A4C8E46.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/Run2015D/SingleMuon/MINIAOD/16Dec2015-v1/10000/00649463-BEA8-E511-9266-0CC47A4D769C.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/Run2015D/SingleMuon/MINIAOD/16Dec2015-v1/10000/00AB7295-C6A8-E511-AEAF-0CC47A4C8E8A.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/Run2015D/SingleMuon/MINIAOD/16Dec2015-v1/10000/00E42A7E-BAA8-E511-B23B-0CC47A4C8E86.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/Run2015D/SingleMuon/MINIAOD/16Dec2015-v1/10000/00F3E56E-BEA8-E511-A6A2-3417EBE722FA.root,\n",
    "                https://xrootd-local.unl.edu:1094//store/user/AGC/datasets/Run2015D/SingleMuon/MINIAOD/16Dec2015-v1/10000/02004059-D9A8-E511-8790-0CC47A78A440.root\n",
    "    Tree: events\n",
    "    FuncADL: \"Where(lambda event: event.nelectron_pt > 5). \\\n",
    "            Select(lambda event: {'electron_pt': event.electron_pt})\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "out_5 = DataBinder(\"config_example_5.yml\").deliver()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=\"3\">\n",
    "\n",
    "The returned file list by ServiceX DataBinder is then passed to coffea framework for a main analysis.\n",
    "\n",
    "Whenever you need new columns or samples, you can modify the configuration file and new ServiceX data delivery requests are submitted only for updated samples. This allows you to maintain your data conveniently.\n",
    "\n",
    "The following code snippet is an example from my coffea analysis code:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "    from coffea.nanoevents import NanoAODSchema\n",
    "    from coffea import processor\n",
    "\n",
    "    run = processor.Runner(processor.FuturesExecutor(workers=30, merging=True),\n",
    "                       format=\"parquet\", schema=NanoAODSchema)\n",
    "    output = run(fileset, \"Events\", fcnc_2lSS())\n",
    "\n",
    "<br>\n",
    "<br>\n",
    "<br>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=\"3\">\n",
    "\n",
    "<span style=\"color:blue\">ServiceX + Coffea + Cabinetry (pyhf) </span> workflow (a.k.a. IRIS-HEP Analysis Grand Challenge workflow) allows an analyzer to perform whole analysis within Python ecosystem. This workflow is not covered in this notebook.\n",
    "\n",
    "Let me advertize the talk tomorrow morning/afternoon/night by Alex Held for a fascinating demo of this workflow."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://indico.cern.ch/event/1150631/timetable/#16-end-to-end-physics-analysis\" target=\"_blank\">\n",
    "    <img src=\"img/agc.png\" height=\"140\"  style=\"float:center\" alt=\"AGC\">\n",
    "</a>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}