From 9b7ce64eb7fab24219bc1baa5b57f8df69d74187 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Brigitta=20Sip=C5=91cz?= <bsipocz@gmail.com>
Date: Wed, 17 Jul 2024 10:57:00 -0700
Subject: [PATCH 1/7] CONT: adding cloud access notebook

---
 tutorials/cloud_access/cloud-access-intro.md | 345 +++++++++++++++++++
 1 file changed, 345 insertions(+)
 create mode 100644 tutorials/cloud_access/cloud-access-intro.md

diff --git a/tutorials/cloud_access/cloud-access-intro.md b/tutorials/cloud_access/cloud-access-intro.md
new file mode 100644
index 00000000..b9f0bfab
--- /dev/null
+++ b/tutorials/cloud_access/cloud-access-intro.md
@@ -0,0 +1,345 @@
+
+# IRSA Cloud Access Introduction
+
+This is the introductory tutorial demonstrating basic python access to the IRSA-curated images and catalogs available in AWS S3 cloud storage buckets.
+
+Learning Goals:
+
+- Find datasets and connection information.
+- Browse buckets.
+- Find an image and retrieve a cutout.
+- Navigate a catalog and perform a basic query.
+
++++
+
+## 1. Cloud basics
+
+### 1.1 Terminology
+
+AWS S3 is an [object store](https://en.wikipedia.org/wiki/Object_storage) where the fundamental entities are "buckets" and "objects".
+Buckets are containers for objects, and objects are blobs of data.
+Users may be more familiar with [filesystem](https://en.wikipedia.org/wiki/File_system) "files" and "directories".
+Files are analogous to objects, and the terms are often used interchangeably.
+While there is no directory-like structure in S3 itself, various layers on top of S3 have implemented functionality to mimic it.
+As a result, users can generally interact using familiar filesystem logic, but may notice subtle differences.
+
+The following S3 terms are also used in this notebook:
+
+- Key: Unique ID for an object, relative to the bucket. Analogous to the full path to a file.
+- Prefix: String of characters at the beginning of the object key. Analogous to the full path to a directory.
+
++++
+
+### 1.2 General access
+
+Most of the common python methods used to read images and catalogs from a local disk can also be pointed at cloud storage buckets.
+This includes methods like Astropy `fits.open` and Pandas `read_parquet`.
+The cloud connection is handled by a separate library, usually [s3fs](https://s3fs.readthedocs.io), [fsspec](https://filesystem-spec.readthedocs.io), or [pyarrow.fs](https://arrow.apache.org/docs/python/api/filesystems.html).
+
+The IRSA buckets are public and access is free.
+Credentials are not required.
+Anonymous connections can be made, often by setting a keyword argument like `anon=True`.
+
++++
+
+## 2. Find datasets and connection information
+
+A listing of the available datasets is maintained on [IRSA's Cloud Access](https://irsa.ipac.caltech.edu/cloud_access/) page.
+It can be used to substitute values for the following variables, which are used throughout this notebook and identified by all caps:
+
+| Cloud Access Variable | Definition |
+| --- | --- |
+| `BUCKET_NAME` | AWS S3 bucket name. |
+| `BUCKET_REGION` | AWS region the bucket is in. |
+| `IMAGES_PREFIX`| S3 prefix to the base of an image set. |
+| `CATALOG_PREFIX` | S3 prefix to the base of a catalog. |
+| `PARQUET_NAME` | Path to the base of the catalog's Parquet dataset, relative to the prefix. |
+
++++
+
+## 3. Imports
+
+Libraries are imported at the top of each section when used.
+This cell will install them if needed:
+
+```{code-cell}
+try:
+    import astropy  # perform image cutouts
+    import hpgeom  # map sky location to catalog partition
+    import matplotlib  # view results
+    import pandas  # query Parquet datasets
+    import pyarrow  # work with Parquet datasets
+    import pyvo  # locate images in the cloud (pyvo>=1.5 required)
+    import s3fs  # browse buckets
+except ImportError:
+    %pip install astropy
+    %pip install hpgeom
+    %pip install pandas
+    %pip install pyarrow
+    %pip install pyvo>=1.5
+    %pip install s3fs
+    %pip install -U matplotlib
+
+# check for pyvo>=1.5 (required for SIA2Service)
+try:
+    import pyvo
+    pyvo.dal.SIA2Service("https://irsa.ipac.caltech.edu/SIA")
+except AttributeError:
+    %pip install --upgrade pyvo
+    # note that you may need to restart the kernel to use the updated package
+```
+
+## 4. Browse buckets
+
+Generally, it is not necessary to look through the buckets.
+The Cloud Access page is the best way to understand what is available.
+However, browsing buckets can sometimes be useful.
+
+The [s3fs](https://s3fs.readthedocs.io) library offers a filesystem-like experience and is demonstrated below.
+Beware that these datasets can contain millions of files or more --
+if calls are taking a long time to return, try narrowing the search by adding additional levels to the prefix.
+
+Users who want to interact with the buckets and objects more directly may instead prefer the [AWS Boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html) library.
+
+```{code-cell}
+import s3fs
+```
+
+```{code-cell}
+# create an S3 client
+s3 = s3fs.S3FileSystem(anon=True)
+```
+
+`ls` the root directory of the Spitzer SEIP Super Mosaics image set:
+
+```{code-cell}
+BUCKET_NAME = "nasa-irsa-spitzer"
+IMAGES_PREFIX = "spitzer/seip/seip_science/images"
+
+s3.ls(f"{BUCKET_NAME}/{IMAGES_PREFIX}/")
+```
+
+Use wildcard (glob) matching to find mosaics.
+This can be slow if looking through a large number of files, so we'll add a few levels to the base prefix to reduce the number of files in the search.
+The additional levels can be found using `ls` recursively or constructed using information on the Cloud Access page.
+
+```{code-cell}
+sub_prefix = "4/0019/40019821/9"
+glob_pattern = "**/*.mosaic.fits"
+
+s3.glob(f"{BUCKET_NAME}/{IMAGES_PREFIX}/{sub_prefix}/{glob_pattern}")
+```
+
+## 5. Find an image and retrieve a cutout
+
+A basic image cutout using [Astropy](https://docs.astropy.org/en/stable/) is demonstrated in section 5.2.
+It requires a bucket name and image key.
+Section 5.1 demonstrates how to query IRSA for this information using [PyVO](https://pyvo.readthedocs.io/en/latest/index.html) and a coordinate search.
+
+Note: There may be a delay between the time when a new dataset is made available in S3 and when IRSA's image services are able to return the new cloud access information.
+In this case, the IRSA URL that is returned by the service can be mapped to the cloud information.
+This is explained further on the Cloud Access page, and also demonstrated in section 5.1 below.
+
+```{code-cell}
+import astropy.io
+import json
+import pyvo
+from astropy import units as u
+from astropy.coordinates import SkyCoord
+from astropy.nddata import Cutout2D
+from astropy.wcs import WCS
+from matplotlib import pyplot as plt
+```
+
+### 5.1 Find an image using PyVO and a coordinate search
+
+```{code-cell}
+coords = SkyCoord("150.01d 2.2d", frame="icrs")
+size = 0.01 * u.deg
+```
+
+For the full list of datasets that can be used with the `collection` parameter in the SIA search below, uncomment the next cell.
+Note that only some of these are currently available in cloud storage.
+To request that additional datasets be made available in cloud storage, please contact IRSA's [Help Desk](https://irsa.ipac.caltech.edu/docs/help_desk.html).
+
+```{code-cell}
+# from astroquery.ipac.irsa import Irsa
+
+# Irsa.list_collections()
+
+# # If `list_collections` is not found,
+# # uncomment the next line to update astroquery, then restart your kernel.
+# # !pip install --upgrade --pre astroquery
+```
+
+Use PyVO to execute a search for Spitzer SEIP Super Mosaics and find cloud access information:
+
+```{code-cell}
+irsa_SIA = pyvo.dal.SIA2Service("https://irsa.ipac.caltech.edu/SIA")
+seip_results = irsa_SIA.search((coords, size), collection="spitzer_seip")
+
+# view cloud access info for the first few files overlapping the search area
+seip_results["cloud_access"][:5]
+```
+
+**Option 1**: Extract the cloud information from the "cloud_access" column for a single mosaic file:
+
+```{code-cell}
+# find the first mosaic file in the results
+# use json to convert the string containing the cloud info to a dictionary
+seip_mosaic_cloud_info = json.loads([i for i in seip_results["cloud_access"] if ".mosaic.fits" in i][0])
+
+# extract
+BUCKET_NAME = seip_mosaic_cloud_info["aws"]["bucket_name"]
+image_key = seip_mosaic_cloud_info["aws"]["key"]
+```
+
+**Option 2**: Construct the cloud information for a single mosaic file from the IRSA URL (useful if you know the dataset is in S3, but results in the "cloud_access" column were empty):
+
+```{code-cell}
+BUCKET_NAME = "nasa-irsa-spitzer"
+IMAGES_PREFIX = "spitzer/seip/seip_science/images"
+
+# find the first mosaic URL in the results
+image_url = [url for url in seip_results["access_url"] if ".mosaic.fits" in url][0]
+
+# construct the key by replacing the first part of the URL with the IMAGES_PREFIX
+# split the URL at the data product ID, the final level of the prefix
+data_product_id = IMAGES_PREFIX.split("/")[-1]
+key_suffix = image_url.split(f"/{data_product_id}/")[-1]
+image_key = f"{IMAGES_PREFIX}/{key_suffix}"
+```
+
+### 5.2 Retrieve a cutout using Astropy
+
+Lazy-load data for better performance.
+The relevant options are enabled by default when using `astropy.io.fits.open` with `fsspec`.
+In addition, use the HDU `section` method in place of the usual `data` to avoid downloading the full data block.
+(See [Obtaining subsets from cloud-hosted FITS files](https://docs.astropy.org/en/stable/io/fits/usage/cloud.html#fits-io-cloud).)
+
+```{code-cell}
+with astropy.io.fits.open(f"s3://{BUCKET_NAME}/{image_key}", fsspec_kwargs={"anon": True}) as hdul:
+    cutout = Cutout2D(hdul[0].section, position=coords, size=size, wcs=WCS(hdul[0].header))
+```
+
+Show the cutout:
+
+```{code-cell}
+plt.imshow(cutout.data, cmap="gray")
+```
+
+## 6. Navigate a catalog and perform a basic query
+
+The catalogs are in Apache Parquet format and partitioned spatially using HEALPix order k=5.
+Parquet datasets can be queried directly using SQL-like logic by applying filters during the file reads.
+For queries that have a spatial constraint, including a filter on the partitioning column can greatly improve performance.
+
+Section 6.1 demonstrates how to view the catalog schema using [PyArrow](https://arrow.apache.org/docs/python/) to find the column names, etc. needed for constructing queries.
+A basic spatial query using [Pandas](https://pandas.pydata.org/docs/) is shown in section 6.2 and includes finding the relevant partitions using [HPGeom](https://hpgeom.readthedocs.io/en/latest/).
+
+```{code-cell}
+import hpgeom
+import pandas as pd
+import pyarrow.dataset
+import pyarrow.fs
+from matplotlib import colors
+```
+
+```{code-cell}
+BUCKET_NAME = "nasa-irsa-wise"
+BUCKET_REGION = "us-west-2"
+
+CATALOG_PREFIX = "wise/allwise/catalogs/p3as_psd/healpix_k5"
+PARQUET_NAME = "wise-allwise.parquet"
+
+parquet_root = f"{BUCKET_NAME}/{CATALOG_PREFIX}/{PARQUET_NAME}"
+
+fs = pyarrow.fs.S3FileSystem(region=BUCKET_REGION, anonymous=True)
+```
+
+### 6.1 View the schema using PyArrow
+
+```{code-cell}
+# load the schema from the "_common_metadata" file
+schema = pyarrow.dataset.parquet_dataset(f"{parquet_root}/_common_metadata", filesystem=fs).schema
+
+# the full schema can be quite large since catalogs often have hundreds of columns
+# but if you do want to look at the entire schema, uncomment the next line
+# schema
+```
+
+Search through the column names to find the HEALPix partitioning columns:
+
+```{code-cell}
+[name for name in schema.names if "healpix" in name]
+# the result shows that both orders k=0 and k=5 are available, but typical use cases only need k=5
+```
+
+Look at a specific column ("ext_flg" will be used below to query for likely stars):
+
+```{code-cell}
+# this will display basic information like name and type
+schema.field("ext_flg")
+```
+
+```{code-cell}
+# units and descriptions are in the field's metadata attribute
+schema.field("ext_flg").metadata
+```
+
+### 6.2 Perform a basic spatial search with Pandas
+
+Query the AllWISE catalog for likely stars within an RA/Dec polygon.
+
+Spatial limits roughly covering the Taurus L1495 star-forming region:
+
+```{code-cell}
+ra_min, ra_max, dec_min, dec_max = 62, 66, 25, 29  # deg
+
+polygon_corners = [(ra_min, dec_min), (ra_min, dec_max), (ra_max, dec_max), (ra_max, dec_min)]
+corners = list(zip(*polygon_corners))  # [(ra values), (dec values)]
+```
+
+Find the partitions (HEALPix pixel indexes) that overlap the polygon:
+
+```{code-cell}
+k = 5
+polygon_pixels = hpgeom.query_polygon(a=corners[0], b=corners[1], nside=hpgeom.order_to_nside(k), inclusive=True)
+```
+
+Query:
+
+```{code-cell}
+results_df = pd.read_parquet(
+    parquet_root,
+    filesystem=fs,
+    # columns to be returned. similar to a SQL SELECT clause.
+    columns=["designation", "w1mpro", "w2mpro", "w3mpro", "w4mpro", f"healpix_k{k}"],
+    # row filters. similar to a SQL WHERE clause.
+    # supported operators: ==, !=, <, >, <=, >=, in, not in
+    # tuple conditions are joined by AND (for OR, use a list of lists)
+    filters=[
+        ("ext_flg", "==", 0),
+        ("ra", ">", ra_min),
+        ("ra", "<", ra_max),
+        ("dec", ">", dec_min),
+        ("dec", "<", dec_max),
+        # include filter on partition column for efficiency. similar to a database index.
+        (f"healpix_k{k}", "in", polygon_pixels),
+    ],
+)
+```
+
+View results on a color-color diagram:
+
+```{code-cell}
+results_df["W1-W2"] = results_df["w1mpro"] - results_df["w2mpro"]
+results_df["W2-W3"] = results_df["w2mpro"] - results_df["w3mpro"]
+results_df.plot.hexbin("W2-W3", "W1-W2", norm=colors.LogNorm(vmin=1, vmax=500))
+```
+
+## About this notebook
+
+- Author: Troy Raen (IRSA Developer) in conjunction with Brigitta Sipőcz, Jessica Krick and the IPAC Science Platform team
+- Updated: 2023-12-22
+- Contact: https://irsa.ipac.caltech.edu/docs/help_desk.html

From e2b271831b298e72973456fab5e07c53fb192177 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Brigitta=20Sip=C5=91cz?= <bsipocz@gmail.com>
Date: Wed, 17 Jul 2024 11:37:21 -0700
Subject: [PATCH 2/7] CONT: adding openuniverse tutorials

---
 ...se2024_roman_simulated_timedomainsurvey.md | 308 ++++++++++++++++++
 ...erse2024_roman_simulated_wideareasurvey.md | 172 ++++++++++
 2 files changed, 480 insertions(+)
 create mode 100644 tutorials/openuniversesims/openuniverse2024_roman_simulated_timedomainsurvey.md
 create mode 100644 tutorials/openuniversesims/openuniverse2024_roman_simulated_wideareasurvey.md

diff --git a/tutorials/openuniversesims/openuniverse2024_roman_simulated_timedomainsurvey.md b/tutorials/openuniversesims/openuniverse2024_roman_simulated_timedomainsurvey.md
new file mode 100644
index 00000000..b74e2f1f
--- /dev/null
+++ b/tutorials/openuniversesims/openuniverse2024_roman_simulated_timedomainsurvey.md
@@ -0,0 +1,308 @@
+---
+jupytext:
+  text_representation:
+    extension: .md
+    format_name: myst
+    format_version: 0.13
+    jupytext_version: 1.16.2
+kernelspec:
+  display_name: science_demo
+  language: python
+  name: conda-env-science_demo-py
+---
+
+# Explore OpenUniverse 2024 Data Preview: Simulated Roman Time Domain Survey Data
+
++++
+
+## Learning Goals:
+
+By the end of this tutorial, you will:
+
+1. learn more about the "observations" that make up the simulated Roman TDS preview.
+2. learn how to find the locations of simulated supernovae in the preview data.
+3. learn how to create aligned cutouts of simulated Roman images.
+4. learn how to make an animated gif from these cutouts.
+
++++
+
+# Import required modules
+
+```{code-cell} ipython3
+# Install libraries if necessary
+!pip install astropy matplotlib numpy pandas pyarrow s3fs scipy
+```
+
+```{code-cell} ipython3
+# Import modules
+import warnings
+
+import astropy.units as u
+import matplotlib.pyplot as plt
+import numpy as np
+import pandas as pd
+from astropy.coordinates import SkyCoord
+from astropy.io import fits
+from astropy.nddata import Cutout2D
+from astropy.nddata.utils import NoOverlapError
+from astropy.table import Table
+from astropy.wcs import WCS, FITSFixedWarning
+from matplotlib import animation
+from scipy.ndimage import rotate
+
+# Needed to access data in the cloud
+import s3fs
+s3 = s3fs.S3FileSystem(anon=True) # create an S3 client
+
+# Filter out the FITSFixedWarning, which is consequenceless and gets thrown every time you deal with a WCS
+# in a Roman openuniverse simulated image using astropy.
+warnings.simplefilter('ignore',category=FITSFixedWarning)
+```
+
+## Define a module to get the date (mjd) of a particular pointing.
+
+```{code-cell} ipython3
+def get_mjd(pointing,
+            obseq_path=f's3://nasa-irsa-simulations/openuniverse2024/roman/preview/RomanTDS/Roman_TDS_obseq_11_6_23.fits'):
+
+    """
+    Retrieve MJD of a given pointing.
+
+    :param pointing: Pointing ID.
+    :type pointing: int
+    :param obseq_path: Path to obseq file Roman_TDS_obseq_11_6_23.fits.
+    :type obseq_path: str, optional
+    :return: MJD of specified pointing.
+    :rtype: float
+    """
+
+    with fits.open(obseq_path, fsspec_kwargs={"anon": True}) as obs:
+        obseq = Table(obs[1].data)
+    mjd = float(obseq['date'][int(pointing)])
+
+    return mjd
+```
+
+## Define a module to create an animated gif from a collection of cutouts.
+
+```{code-cell} ipython3
+def animate_stamps(stamps,savepath,no_whitespace=True,
+                   labels=[],labelxy=(0.05,0.95),
+                   **kwargs):
+    """
+    Make an animation of a sequence of image stamps.
+
+    :param stamps: Must be in chronological order.
+    :type stamps: List of stamps from get_stamps or get_object_instances.
+    :param savepath: Path to save gif.
+    :type savepath: str
+    """
+
+    if no_whitespace:
+        with_whitespace = np.invert(np.any((np.isnan(np.array(stamps))), axis=(1,2))) # NOTE: Your first axis (first indexing value) should return one stamp. e.g. stamps[0] is the first stamp.
+        idx_whitespace = np.where(with_whitespace)[0]
+        stamps = np.array(stamps)[idx_whitespace]
+        if len(labels) != 0:
+            labels = np.array(labels)[idx_whitespace]
+
+    fig, ax = plt.subplots(figsize=(5,5))
+    fig.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=None, hspace=None)
+    plt.xticks([])
+    plt.yticks([])
+
+    im = ax.imshow(stamps[0], animated=True)
+
+    if len(labels) != 0:
+        txt = ax.text(labelxy[0],labelxy[1],labels[0],animated=True,color='white',transform=ax.transAxes,va='top',ha='left',**kwargs)
+
+    def animate(i):
+        im.set_array(stamps[i])
+        if len(labels) != 0:
+            txt.set_text(labels[i])
+
+            return [im] + [txt]
+        else:
+            return [im]
+
+    writer = animation.PillowWriter()
+    anim = animation.FuncAnimation(fig, animate, interval=600, frames=len(stamps))
+    anim.save(savepath, writer=writer)
+```
+
+## Read in the Observation Sequence File to learn more about the "observations" that make up the simulated Roman Time Domain Survey.
+
+```{code-cell} ipython3
+# Read in the (simulated) Observation Sequence File.
+
+BUCKET_NAME = 'nasa-irsa-simulations'
+ROMAN_PREFIX = 'openuniverse2024/roman/preview'
+
+ROMAN_TDS_PATH = f'{ROMAN_PREFIX}/RomanTDS'
+FILENAME = 'Roman_TDS_obseq_11_6_23.fits'
+OBSEQ_PATH = f's3://{BUCKET_NAME}/{ROMAN_TDS_PATH}/{FILENAME}'
+
+obseq_hdu = fits.open(OBSEQ_PATH, fsspec_kwargs={"anon": True})
+obseq = pd.DataFrame(obseq_hdu[1].data)
+
+print(obseq)
+```
+
+## What is the spatial and temporal coverage of the openuniverse2024 Roman TDS data preview?
+
+```{code-cell} ipython3
+# Find the ranges of RA, Dec, and date listed in the observation sequence file.
+
+ra_min, dec_min = obseq[['ra','dec']].min()
+ra_max, dec_max = obseq[['ra','dec']].max()
+mjd_min = obseq['date'].min()
+mjd_max = obseq['date'].max()
+
+print("ra_min, ra_max:", ra_min, ra_max)
+print("mjd_min, mjd_max:", mjd_min, mjd_max)
+```
+
+## Read in the Supernova Analysis (SNANA) file.
+
+```{code-cell} ipython3
+parquet_file = f's3://{BUCKET_NAME}/{ROMAN_PREFIX}/roman_rubin_cats_v1.1.2_faint/snana_10307.parquet'
+transients = pd.read_parquet(parquet_file, filesystem=s3)
+```
+
+## Let's find a relatively nearby SN Ia that lies within the region of the data preview. 
+
+```{code-cell} ipython3
+#List the unique models in the SNANA file.
+unique_models = pd.Series(transients['model_name']).drop_duplicates().tolist()
+unique_models
+```
+
+```{code-cell} ipython3
+# Most of the models are non SNIa (NON1ASED).
+# Choose only the SNIa
+sn1a = transients[transients['model_name'] == 'SALT3.NIR_WAVEEXT'] # SNe Ia only.
+print('Number of SN1a in SNANA file: ', len(sn1a))
+```
+
+```{code-cell} ipython3
+# Choose the SNIa that overlap with the spatial extent of the OpenUniverse2024 Roman TDS data preview.
+ra_mask = np.logical_and(sn1a['ra'] > ra_min, sn1a['ra'] < ra_max)
+dec_mask = np.logical_and(sn1a['dec'] > dec_min, sn1a['dec'] < dec_max)
+mjd_mask = np.logical_and(sn1a['start_mjd'] > mjd_min, sn1a['end_mjd'] < mjd_max)
+all_mask = np.logical_and.reduce((ra_mask,dec_mask,mjd_mask))
+preview_sn1a = sn1a[all_mask]
+print('Number of SNIa in OpenUniverse2024 data preview:', len(preview_sn1a))
+```
+
+```{code-cell} ipython3
+# Choose the SNIa in the data preview that are nearby, at redshifts less than 0.7.
+nearby_preview_sn1a = preview_sn1a[preview_sn1a['z_CMB'] < 0.7]
+print('Number of nearby SNIa in OpenUniverse2024 data preview:', len(nearby_preview_sn1a))
+```
+
+```{code-cell} ipython3
+# Let's choose SN 20000808.
+oid = 20000808
+chosen_object = nearby_preview_sn1a[nearby_preview_sn1a['id'] == oid]
+ra = chosen_object.get('ra')
+dec = chosen_object.get('dec')
+ra, dec = 9.619282, -44.313894
+coord = SkyCoord(ra*u.deg, dec*u.deg)
+```
+
+## Read in the auxiliary file that lists the simulated Roman TDS images covering the chosen SNIa.
+
+```{code-cell} ipython3
+# The auxiliary file contains all the images this thing is in.
+# If you need to download this file, see https://irsa.ipac.caltech.edu/docs/notebooks/.
+csvfile = './openuniverse2024_roman_demodata_20000808_instances.csv'
+instances = pd.read_csv(csvfile, usecols=['filter','pointing','sca'])
+instances
+```
+
+## Create cutouts of the chosen SNIa in the band of your choice.
+
+```{code-cell} ipython3
+band = 'R062'
+instances = instances[instances['filter'] == band]
+```
+
+```{code-cell} ipython3
+#Make the cutouts; this will take a couple of minutes.
+stamps = []
+mjd = []
+for i, row in enumerate(instances.itertuples()):
+    band, pointing, sca = row[1], row[2], row[3]
+    imgpath = f's3://{BUCKET_NAME}/{ROMAN_TDS_PATH}/images/simple_model/{band}/{pointing}/Roman_TDS_simple_model_{band}_{pointing}_{sca}.fits.gz'
+    print(imgpath)
+    with fits.open(imgpath, fsspec_kwargs={"anon": True}) as hdu:
+        img = hdu[1].data
+        header = hdu[0].header
+        wcs = WCS(header)
+        x, y = wcs.world_to_pixel(coord)
+
+        # Manually rotate the images so they are all aligned.
+        CDmat = np.array([header['CD1_1'], header['CD1_2'],
+                          header['CD2_1'], header['CD2_2']]).reshape(2,2)
+
+        orientation = hdu[0].header['ORIENTAT']
+
+        # These chips are "flipped".
+        if sca % 3 == 0:
+            orientation += 180
+
+        # Build rotation matrix.
+        CD1_1_rot = np.cos(-orientation*np.pi/180)
+        CD1_2_rot = -np.sin(-orientation*np.pi/180)
+        CD2_1_rot = np.sin(-orientation*np.pi/180)
+        CD2_2_rot = np.cos(-orientation*np.pi/180)
+
+        RotMat = np.array([CD1_1_rot, CD1_2_rot,
+                          CD2_1_rot, CD2_2_rot]).reshape(2,2)
+
+        RotMat_inv = np.array([CD1_1_rot, -CD1_2_rot,
+                              -CD2_1_rot, CD2_2_rot]).reshape(2,2)
+
+        # Apply rotation to the CDi_j header keywords.
+        CDmat_rot = np.dot(CDmat,RotMat_inv)
+
+        # Update header.
+        header['CD1_1'], header['CD1_2'] = CDmat_rot[0]
+        header['CD2_1'], header['CD2_2'] = CDmat_rot[1]
+        header['ORIENTAT'] -= orientation
+
+        # Rotate the image.
+        rot_img = rotate(img,angle=orientation,reshape=False,cval=np.nan)
+        hdu[1].data = rot_img
+
+        rot_wcs = WCS(header)
+
+        try:
+            # Make cutout around SN Ia location.
+            cutout = Cutout2D(rot_img,coord,100,wcs=rot_wcs,mode='partial')
+            stamps.append(cutout.data)
+            mjd.append(get_mjd(pointing))
+        except NoOverlapError:
+            pass
+```
+
+## Make an animated gif out of the cutouts.
+
+```{code-cell} ipython3
+savepath = f'SN{oid}.gif'
+savepath
+animate_stamps(stamps,savepath,labels=mjd)
+```
+
+![animated gif](SN20000808.gif)
+
++++
+
+## About this notebook
+
+- Author: Lauren Aldoroty (laurenaldoroty@gmail.com) with minor subsequent modifications to match repository style
+- Contact: https://irsa.ipac.caltech.edu/docs/help_desk.html
+- Updated: 2024-06-10
+
+```{code-cell} ipython3
+
+```
diff --git a/tutorials/openuniversesims/openuniverse2024_roman_simulated_wideareasurvey.md b/tutorials/openuniversesims/openuniverse2024_roman_simulated_wideareasurvey.md
new file mode 100644
index 00000000..8f6c9993
--- /dev/null
+++ b/tutorials/openuniversesims/openuniverse2024_roman_simulated_wideareasurvey.md
@@ -0,0 +1,172 @@
+---
+jupytext:
+  text_representation:
+    extension: .md
+    format_name: myst
+    format_version: 0.13
+    jupytext_version: 1.16.2
+kernelspec:
+  display_name: science_demo
+  language: python
+  name: conda-env-science_demo-py
+---
+
+# Explore OpenUniverse 2024 Data Preview: Simulated Roman Coadds
+
++++
+
+## Learning Goals
+
+By the end of this tutorial, you will learn how to do the following:
+
+- Identify the row and column that contains a particular ra, dec coordinate.
+- Browse the bucket containing the simulated Roman coadds.
+- Identify a simulated Roman coadd by filter, row, and column.
+- Take a closer look at a simulated coadd file.
+
+```{code-cell} ipython3
+# Install libraries if necessary
+!pip install astropy numpy s3fs
+```
+
+```{code-cell} ipython3
+#Import modules
+from astropy.io import fits
+import numpy as np
+import s3fs  # browse buckets
+```
+
+## Identify the row and column that contains a particular ra, dec coordinate.
+
+The full simulation covers a 1 deg x 1 deg area centered around RA = 9.5 deg and 
+Dec = -44.1 deg. This region is divided into 1296 blocks (36 rows and 36 columns), 
+each 100 arcsec across.
+
+The data preview presented here covers the central 20x20 arcmin, corresponding to 144 blocks (12 rows and 12 columns).
+
+```{code-cell} ipython3
+#Choose an RA, DEC of interest.
+ra = 9.5981595
+dec = -44.2026950
+
+#Centers of data preview blocks. Do not alter.
+ra_block_centers = np.array([9.76330352298415, 9.724522605135252, 9.68574158906671,
+                    9.646960496603766, 9.608179349571955, 9.56939816979703,
+                    9.530616979104877, 9.491835799321422, 9.453054652272561,
+                    9.414273559784032, 9.375492543681393, 9.336711625789874])
+dec_block_centers = np.array([-44.252584927082495, -44.22480733304182, -44.197029724175756,
+                              -44.16925210374898, -44.14147447502621, -44.11369684127218,
+                              44.08591920575162, -44.05814157172923, -44.03036394246976,
+                              -44.0025863212379, -43.974808711298394, -43.94703111591591])
+
+ra_difference_array = np.absolute(ra_block_centers-ra)
+ra_block_centers_index = ra_difference_array.argmin()
+closest_ra_center = ra_block_centers[ra_block_centers_index]
+ra_dist = 3600. * ra_difference_array[ra_block_centers_index]
+if ra_dist > 50:
+    print("Chosen ra not covered by OpenUniverse 2024 data preview simulated Roman coadds")
+else:
+    COLUMN = ra_block_centers_index + 12
+    print("COLUMN:", COLUMN)
+    print("")
+
+dec_difference_array = np.absolute(dec_block_centers-dec)
+dec_block_centers_index = dec_difference_array.argmin()
+closest_dec_center = dec_block_centers[dec_block_centers_index]
+dec_dist = 3600. * dec_difference_array[dec_block_centers_index]
+
+if dec_dist > 50:
+    print("Chosen dec not covered by OpenUniverse 2024 data preview simulated Roman coadds")
+else:
+    ROW = dec_block_centers_index + 12
+    print("ROW:", ROW)
+```
+
+## Browse the bucket containing the simulated Roman coadds.
+
+```{code-cell} ipython3
+s3 = s3fs.S3FileSystem(anon=True) # create an S3 client
+
+BUCKET_NAME = "nasa-irsa-simulations"
+ROMAN_PREFIX = "openuniverse2024/roman/preview"
+COADD_PATH = f"{ROMAN_PREFIX}/RomanWAS/images/coadds"
+
+s3.ls(f"{BUCKET_NAME}/{COADD_PATH}")
+```
+
+## Identify a Roman simulated coadd by filter, row, and column.
+
+A simulated coadd can be uniquely identified by filter, row, and column.
+
+```{code-cell} ipython3
+#Choose a filter, row, and column
+FILTER = 'H158' #Filters F184, H158, J129, K213, and Y106 are available in the data preview.
+#ROW = 12 #Rows 12-23 are available in the data preview.
+#COLUMN = 12 #Columns 12-23 are available in the data preview.
+
+#Construct the coadd filename from the chosen filter, row, and column.
+filename_root = f"prod_{FILTER[0]}_{COLUMN}_{ROW}_map.fits"
+
+#Construct the full coadd path from the chosen filter, row, and column.
+s3_uri = f"s3://{BUCKET_NAME}/{COADD_PATH}/{FILTER}/Row{ROW}/{filename_root}"
+
+#List this filename to make sure it is found.
+s3.ls(s3_uri)
+```
+
+## Take a closer look at the simulated coadd file you identified.
+
+```{code-cell} ipython3
+#Show a summary of extensions for this file.
+
+with fits.open(s3_uri, fsspec_kwargs={"anon": True}) as hdul:
+    hdul.info()
+```
+
+The Primary HDU for the coadded image is a cube with 15 layers, i.e., its shape is 1x15x2688x2688. The layers are as follows:
+
+0 = simulated "Science" image (Roman+Rubin simulation, units of e/(0.11 arcsec)^2/exposure)
+
+1 = lab noise: based on dark frames from the April 2023 test, masked at 3 e/p/s. Units: e/(0.11 arcsec)^2/s
+
+2 = GalSim stars, on HEALPix resolution 14 grid, normalized to total flux of 1
+
+3 = noisy stars, on HEALPix resolution 14 grid, normalized to total flux of 2.4e5 e with self-Poisson noise, including 86 e^2/input pixel background variance
+
+4 = stars, on HEALPix resolution 14 grid, total flux 1, but on in only one of the passes (to test transient response)
+
+5 = stars, on HEALPix resolution 14 grid, with total flux that varies by 5% from center to edge of the focal plane (to test what happens when the filter bandpass varies; 5% is highly exaggerated)
+
+6 = GalSim extended objects, on HEALPix resolution 14 grid, right now exponential profiles. The scale radius is log-distributed between 0.125 and 0.500 arcsec, and the ellipticity (g1,g2) is uniformly distributed in the disc of radius 0.5, i.e., g1^2+g2^2<0.5^2.
+
+7,8,9 = same objects as layer 6, but with applied shear of 0.02. The shear orientations are spaced by 60° in tangent vector space, so that in the (g1,g2)-space they are spaced by 120° and can be used for finite differences. Specifically, the directions are: layer 7 -> in East-West direction (shear PA = 270°). (g1,g2) = (0.02,0) layer 8 -> in NNW-SSE direction (shear PA = 330°). (g1,g2) = (-0.02/2,0.02√3/2) layer 9 -> in NNE-SSW direction (shear PA = 30°). (g1,g2) = (-0.02/2,-0.02√3/2)
+
+10 = coadded 1/f noise map, normalized to variance per ln f of 1
+
+11,12,13,14 = coadded white noise maps, different seeds, normalized to variance of 1 in each input pixel
+
+The following HDUs contain additional information:
+
+CONFIG = the configuration file
+
+INDATA = the input images used, as a binary table. The columns are: obsid (int32) -> observation ID sca (int16) -> SCA (1 through 18, inclusive) ra (float64) -> right ascension of pointing center in degrees dec (float64) -> declination of pointing center in degrees pa (float64) -> position angle of pointing in degrees valid (logical) -> input science data file is valid (should be True)
+
+INWEIGHT = the mean input weights for how much each 1.25x1.25 arcsec postage stamp depends on each input exposure. The shape is 1 x Nin x 84 x 84, where Nin is the number of input images listed in INDATA. Note that each postage stamp is 32 output pixels, so 84x32=2688.
+
+If summed on axis 1, this will normally be something close to 1. Deviations of ~10% are common, due to plate scale variations and the normalization issues introduced by diffraction spikes.
+
+INWEIGHTFLAT = a reshape of INWEIGHT suitable for display as a single image in a viewer such as DS9. The contributions from the Nin input exposures are rearranged into a 1 x 84 x (N_in*84) array.
+
+FIDELITY, SIGMA, INWTSUM, EFFCOVER = maps of U_alpha/C, S_alpha, sum_i T_{alpha i}, and the effective coverage as rescaled int16 maps. See Rowe et al. (2011) for details on the definitions of these quantities. The comment in the 'UNIT' keyword indicates how to rescale these.
+
++++
+
+## About this notebook
+
+- Author: Vandana Desai (Science Lead, IRSA) in conjunction with the IPAC Science Platform team
+- Contact: https://irsa.ipac.caltech.edu/docs/help_desk.html
+- Updated: 2024-06-10
+
+```{code-cell} ipython3
+
+```

From 48512bc8b593d5a2e5b6dd860af2a4540af4271c Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Brigitta=20Sipo=CC=8Bcz?= <bsipocz@gmail.com>
Date: Wed, 17 Jul 2024 11:53:28 -0700
Subject: [PATCH 3/7] CONT: fixing build issues for new tutorials

---
 tutorials/cloud_access/cloud-access-intro.md  | 58 +++++++++++--------
 ...2024_roman_demodata_20000808_instances.csv | 40 +++++++++++++
 ...se2024_roman_simulated_timedomainsurvey.md |  4 +-
 ...erse2024_roman_simulated_wideareasurvey.md |  6 +-
 4 files changed, 80 insertions(+), 28 deletions(-)
 create mode 100644 tutorials/openuniversesims/openuniverse2024_roman_demodata_20000808_instances.csv

diff --git a/tutorials/cloud_access/cloud-access-intro.md b/tutorials/cloud_access/cloud-access-intro.md
index b9f0bfab..ccf79c05 100644
--- a/tutorials/cloud_access/cloud-access-intro.md
+++ b/tutorials/cloud_access/cloud-access-intro.md
@@ -1,3 +1,15 @@
+---
+jupytext:
+  text_representation:
+    extension: .md
+    format_name: myst
+    format_version: 0.13
+    jupytext_version: 1.16.2
+kernelspec:
+  display_name: Python 3 (ipykernel)
+  language: python
+  name: python3
+---
 
 # IRSA Cloud Access Introduction
 
@@ -62,7 +74,7 @@ It can be used to substitute values for the following variables, which are used
 Libraries are imported at the top of each section when used.
 This cell will install them if needed:
 
-```{code-cell}
+```{code-cell} ipython3
 try:
     import astropy  # perform image cutouts
     import hpgeom  # map sky location to catalog partition
@@ -101,18 +113,18 @@ if calls are taking a long time to return, try narrowing the search by adding ad
 
 Users who want to interact with the buckets and objects more directly may instead prefer the [AWS Boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html) library.
 
-```{code-cell}
+```{code-cell} ipython3
 import s3fs
 ```
 
-```{code-cell}
+```{code-cell} ipython3
 # create an S3 client
 s3 = s3fs.S3FileSystem(anon=True)
 ```
 
 `ls` the root directory of the Spitzer SEIP Super Mosaics image set:
 
-```{code-cell}
+```{code-cell} ipython3
 BUCKET_NAME = "nasa-irsa-spitzer"
 IMAGES_PREFIX = "spitzer/seip/seip_science/images"
 
@@ -123,7 +135,7 @@ Use wildcard (glob) matching to find mosaics.
 This can be slow if looking through a large number of files, so we'll add a few levels to the base prefix to reduce the number of files in the search.
 The additional levels can be found using `ls` recursively or constructed using information on the Cloud Access page.
 
-```{code-cell}
+```{code-cell} ipython3
 sub_prefix = "4/0019/40019821/9"
 glob_pattern = "**/*.mosaic.fits"
 
@@ -140,7 +152,7 @@ Note: There may be a delay between the time when a new dataset is made available
 In this case, the IRSA URL that is returned by the service can be mapped to the cloud information.
 This is explained further on the Cloud Access page, and also demonstrated in section 5.1 below.
 
-```{code-cell}
+```{code-cell} ipython3
 import astropy.io
 import json
 import pyvo
@@ -153,7 +165,7 @@ from matplotlib import pyplot as plt
 
 ### 5.1 Find an image using PyVO and a coordinate search
 
-```{code-cell}
+```{code-cell} ipython3
 coords = SkyCoord("150.01d 2.2d", frame="icrs")
 size = 0.01 * u.deg
 ```
@@ -162,7 +174,7 @@ For the full list of datasets that can be used with the `collection` parameter i
 Note that only some of these are currently available in cloud storage.
 To request that additional datasets be made available in cloud storage, please contact IRSA's [Help Desk](https://irsa.ipac.caltech.edu/docs/help_desk.html).
 
-```{code-cell}
+```{code-cell} ipython3
 # from astroquery.ipac.irsa import Irsa
 
 # Irsa.list_collections()
@@ -174,7 +186,7 @@ To request that additional datasets be made available in cloud storage, please c
 
 Use PyVO to execute a search for Spitzer SEIP Super Mosaics and find cloud access information:
 
-```{code-cell}
+```{code-cell} ipython3
 irsa_SIA = pyvo.dal.SIA2Service("https://irsa.ipac.caltech.edu/SIA")
 seip_results = irsa_SIA.search((coords, size), collection="spitzer_seip")
 
@@ -184,7 +196,7 @@ seip_results["cloud_access"][:5]
 
 **Option 1**: Extract the cloud information from the "cloud_access" column for a single mosaic file:
 
-```{code-cell}
+```{code-cell} ipython3
 # find the first mosaic file in the results
 # use json to convert the string containing the cloud info to a dictionary
 seip_mosaic_cloud_info = json.loads([i for i in seip_results["cloud_access"] if ".mosaic.fits" in i][0])
@@ -196,7 +208,7 @@ image_key = seip_mosaic_cloud_info["aws"]["key"]
 
 **Option 2**: Construct the cloud information for a single mosaic file from the IRSA URL (useful if you know the dataset is in S3, but results in the "cloud_access" column were empty):
 
-```{code-cell}
+```{code-cell} ipython3
 BUCKET_NAME = "nasa-irsa-spitzer"
 IMAGES_PREFIX = "spitzer/seip/seip_science/images"
 
@@ -217,14 +229,14 @@ The relevant options are enabled by default when using `astropy.io.fits.open` wi
 In addition, use the HDU `section` method in place of the usual `data` to avoid downloading the full data block.
 (See [Obtaining subsets from cloud-hosted FITS files](https://docs.astropy.org/en/stable/io/fits/usage/cloud.html#fits-io-cloud).)
 
-```{code-cell}
+```{code-cell} ipython3
 with astropy.io.fits.open(f"s3://{BUCKET_NAME}/{image_key}", fsspec_kwargs={"anon": True}) as hdul:
     cutout = Cutout2D(hdul[0].section, position=coords, size=size, wcs=WCS(hdul[0].header))
 ```
 
 Show the cutout:
 
-```{code-cell}
+```{code-cell} ipython3
 plt.imshow(cutout.data, cmap="gray")
 ```
 
@@ -237,7 +249,7 @@ For queries that have a spatial constraint, including a filter on the partitioni
 Section 6.1 demonstrates how to view the catalog schema using [PyArrow](https://arrow.apache.org/docs/python/) to find the column names, etc. needed for constructing queries.
 A basic spatial query using [Pandas](https://pandas.pydata.org/docs/) is shown in section 6.2 and includes finding the relevant partitions using [HPGeom](https://hpgeom.readthedocs.io/en/latest/).
 
-```{code-cell}
+```{code-cell} ipython3
 import hpgeom
 import pandas as pd
 import pyarrow.dataset
@@ -245,7 +257,7 @@ import pyarrow.fs
 from matplotlib import colors
 ```
 
-```{code-cell}
+```{code-cell} ipython3
 BUCKET_NAME = "nasa-irsa-wise"
 BUCKET_REGION = "us-west-2"
 
@@ -259,7 +271,7 @@ fs = pyarrow.fs.S3FileSystem(region=BUCKET_REGION, anonymous=True)
 
 ### 6.1 View the schema using PyArrow
 
-```{code-cell}
+```{code-cell} ipython3
 # load the schema from the "_common_metadata" file
 schema = pyarrow.dataset.parquet_dataset(f"{parquet_root}/_common_metadata", filesystem=fs).schema
 
@@ -270,19 +282,19 @@ schema = pyarrow.dataset.parquet_dataset(f"{parquet_root}/_common_metadata", fil
 
 Search through the column names to find the HEALPix partitioning columns:
 
-```{code-cell}
+```{code-cell} ipython3
 [name for name in schema.names if "healpix" in name]
 # the result shows that both orders k=0 and k=5 are available, but typical use cases only need k=5
 ```
 
 Look at a specific column ("ext_flg" will be used below to query for likely stars):
 
-```{code-cell}
+```{code-cell} ipython3
 # this will display basic information like name and type
 schema.field("ext_flg")
 ```
 
-```{code-cell}
+```{code-cell} ipython3
 # units and descriptions are in the field's metadata attribute
 schema.field("ext_flg").metadata
 ```
@@ -293,7 +305,7 @@ Query the AllWISE catalog for likely stars within an RA/Dec polygon.
 
 Spatial limits roughly covering the Taurus L1495 star-forming region:
 
-```{code-cell}
+```{code-cell} ipython3
 ra_min, ra_max, dec_min, dec_max = 62, 66, 25, 29  # deg
 
 polygon_corners = [(ra_min, dec_min), (ra_min, dec_max), (ra_max, dec_max), (ra_max, dec_min)]
@@ -302,14 +314,14 @@ corners = list(zip(*polygon_corners))  # [(ra values), (dec values)]
 
 Find the partitions (HEALPix pixel indexes) that overlap the polygon:
 
-```{code-cell}
+```{code-cell} ipython3
 k = 5
 polygon_pixels = hpgeom.query_polygon(a=corners[0], b=corners[1], nside=hpgeom.order_to_nside(k), inclusive=True)
 ```
 
 Query:
 
-```{code-cell}
+```{code-cell} ipython3
 results_df = pd.read_parquet(
     parquet_root,
     filesystem=fs,
@@ -332,7 +344,7 @@ results_df = pd.read_parquet(
 
 View results on a color-color diagram:
 
-```{code-cell}
+```{code-cell} ipython3
 results_df["W1-W2"] = results_df["w1mpro"] - results_df["w2mpro"]
 results_df["W2-W3"] = results_df["w2mpro"] - results_df["w3mpro"]
 results_df.plot.hexbin("W2-W3", "W1-W2", norm=colors.LogNorm(vmin=1, vmax=500))
diff --git a/tutorials/openuniversesims/openuniverse2024_roman_demodata_20000808_instances.csv b/tutorials/openuniversesims/openuniverse2024_roman_demodata_20000808_instances.csv
new file mode 100644
index 00000000..a6a3610b
--- /dev/null
+++ b/tutorials/openuniversesims/openuniverse2024_roman_demodata_20000808_instances.csv
@@ -0,0 +1,40 @@
+,filter,pointing,sca
+14,R062,407,7
+19,R062,4257,10
+8,R062,4647,6
+6,R062,6182,5
+27,R062,8112,15
+31,R062,9652,17
+0,R062,11197,1
+1,R062,11582,1
+17,R062,15037,8
+23,R062,16962,11
+20,R062,18502,10
+9,R062,18507,6
+10,R062,20812,6
+37,R062,21592,18
+32,R062,21977,17
+28,R062,22357,15
+33,R062,23512,17
+2,R062,25442,1
+25,R062,27367,13
+15,R062,28512,7
+21,R062,32362,10
+11,R062,32752,6
+7,R062,34287,5
+29,R062,36217,15
+34,R062,37757,17
+3,R062,39302,1
+4,R062,39687,1
+18,R062,43142,8
+24,R062,45067,11
+22,R062,46607,10
+12,R062,46612,6
+13,R062,48917,6
+38,R062,49697,18
+35,R062,50082,17
+30,R062,50462,15
+36,R062,51617,17
+5,R062,53547,1
+26,R062,55472,13
+16,R062,56617,7
diff --git a/tutorials/openuniversesims/openuniverse2024_roman_simulated_timedomainsurvey.md b/tutorials/openuniversesims/openuniverse2024_roman_simulated_timedomainsurvey.md
index b74e2f1f..5d8bdb61 100644
--- a/tutorials/openuniversesims/openuniverse2024_roman_simulated_timedomainsurvey.md
+++ b/tutorials/openuniversesims/openuniverse2024_roman_simulated_timedomainsurvey.md
@@ -8,7 +8,7 @@ jupytext:
 kernelspec:
   display_name: science_demo
   language: python
-  name: conda-env-science_demo-py
+  name: python3
 ---
 
 # Explore OpenUniverse 2024 Data Preview: Simulated Roman Time Domain Survey Data
@@ -168,7 +168,7 @@ parquet_file = f's3://{BUCKET_NAME}/{ROMAN_PREFIX}/roman_rubin_cats_v1.1.2_faint
 transients = pd.read_parquet(parquet_file, filesystem=s3)
 ```
 
-## Let's find a relatively nearby SN Ia that lies within the region of the data preview. 
+## Let's find a relatively nearby SN Ia that lies within the region of the data preview.
 
 ```{code-cell} ipython3
 #List the unique models in the SNANA file.
diff --git a/tutorials/openuniversesims/openuniverse2024_roman_simulated_wideareasurvey.md b/tutorials/openuniversesims/openuniverse2024_roman_simulated_wideareasurvey.md
index 8f6c9993..d4b9ca4c 100644
--- a/tutorials/openuniversesims/openuniverse2024_roman_simulated_wideareasurvey.md
+++ b/tutorials/openuniversesims/openuniverse2024_roman_simulated_wideareasurvey.md
@@ -8,7 +8,7 @@ jupytext:
 kernelspec:
   display_name: science_demo
   language: python
-  name: conda-env-science_demo-py
+  name: python3
 ---
 
 # Explore OpenUniverse 2024 Data Preview: Simulated Roman Coadds
@@ -38,8 +38,8 @@ import s3fs  # browse buckets
 
 ## Identify the row and column that contains a particular ra, dec coordinate.
 
-The full simulation covers a 1 deg x 1 deg area centered around RA = 9.5 deg and 
-Dec = -44.1 deg. This region is divided into 1296 blocks (36 rows and 36 columns), 
+The full simulation covers a 1 deg x 1 deg area centered around RA = 9.5 deg and
+Dec = -44.1 deg. This region is divided into 1296 blocks (36 rows and 36 columns),
 each 100 arcsec across.
 
 The data preview presented here covers the central 20x20 arcmin, corresponding to 144 blocks (12 rows and 12 columns).

From 707d365c882b57b164fedf7c364daebdf20cde43 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Brigitta=20Sip=C5=91cz?= <bsipocz@gmail.com>
Date: Thu, 18 Jul 2024 12:53:11 -0700
Subject: [PATCH 4/7] CONT: making headings more consistent

---
 index.md                                      | 63 ++++++++++++++++++-
 .../irsa-sia-examples/sia_2mass_allsky.md     |  4 +-
 .../irsa-sia-examples/sia_allwise_atlas.md    |  4 +-
 tutorials/irsa-sia-examples/sia_cosmos.md     |  2 +-
 tutorials/irsa-sia-examples/siav2_seip.md     |  2 +-
 ...se2024_roman_simulated_timedomainsurvey.md |  4 +-
 ...erse2024_roman_simulated_wideareasurvey.md |  4 +-
 7 files changed, 71 insertions(+), 12 deletions(-)

diff --git a/index.md b/index.md
index 28dc9c9c..072df7fa 100644
--- a/index.md
+++ b/index.md
@@ -7,7 +7,11 @@ They cover topics like querying IRSA, working with catalogs in Parquet format, a
 
 ## Accessing IRSA archive holdings
 
-### Images
+### Image Thumbnailes
+
+These notebooks show how to query IRSA's image services, inspect the results, and download and visualize images.
+
+They use the tools: PyVO, Astropy.
 
 ```{toctree}
 ---
@@ -21,8 +25,46 @@ tutorials/irsa-sia-examples/siav2_seip
 
 ```
 
+<!---
+### Catalogs
+
+```{toctree}
+---
+maxdepth: 1
+---
+
+
+```
+
+## Visualizations
+```{toctree}
+---
+maxdepth: 1
+---
+
+```
+-->
+
+
 ## IRSA in the cloud
 
+This notebook demonstrates basic access to the IRSA-curated datasets available in AWS S3 cloud storage buckets.
+
+It uses the tools: Pandas, PyArrow, Astropy, Astroquery, PyVO, S3FS
+
+```{toctree}
+---
+maxdepth: 1
+---
+
+tutorials/cloud_access/cloud-access-intro
+```
+
+### Catalogs
+
+This notebook shows examples for the Parquet version of the AllWISE Source Catalog, located in AWS S3 cloud storage.
+It uses the tools: Pandas, PyArrow, Astropy
+
 
 ```{toctree}
 ---
@@ -30,7 +72,21 @@ maxdepth: 1
 ---
 
 tutorials/parquet-catalog-demos/wise-allwise-catalog-demo
-tutorials/cosmosims/CosmoDC2_Parquet
+
+```
+
+### Explore OpenUniverse 2024 Data Preview
+
+These notebooks explore simulared Roman observation stored in AWS S3 cloud storage.
+They use the tools: Pandas, Astropy, S3FS, matplotlib, NumPy
+
+```{toctree}
+---
+maxdepth: 1
+---
+
+tutorials/openuniversesims/openuniverse2024_roman_simulated_timedomainsurvey
+tutorials/openuniversesims/openuniverse2024_roman_simulated_wideareasurvey
 ```
 
 ## Generally useful techniques
@@ -41,4 +97,5 @@ maxdepth: 1
 ---
 
 tutorials/parallelize/Parallelize_Convolution
-```
\ No newline at end of file
+
+```
diff --git a/tutorials/irsa-sia-examples/sia_2mass_allsky.md b/tutorials/irsa-sia-examples/sia_2mass_allsky.md
index 4c1c5cb6..2a53d7e9 100644
--- a/tutorials/irsa-sia-examples/sia_2mass_allsky.md
+++ b/tutorials/irsa-sia-examples/sia_2mass_allsky.md
@@ -11,7 +11,7 @@ kernelspec:
   name: python3
 ---
 
-# Image Thumbnails from 2MASS All-Sky Atlas
+# 2MASS All-Sky Atlas
 
 This notebook tutorial demonstrates the process of querying IRSA's Simple Image Access (SIA) service for the 2MASS All-Sky Atlas, making a cutout image (thumbnail), and displaying the cutout.
 
@@ -150,7 +150,7 @@ wcs = cutout.wcs
 fig = plt.figure()
 
 ax = fig.add_subplot(1, 1, 1, projection=wcs)
-ax.imshow(cutout.data, cmap='gray_r', origin='lower', 
+ax.imshow(cutout.data, cmap='gray_r', origin='lower',
           vmax = 1000)
 ax.scatter(ra, dec, transform=ax.get_transform('fk5'), s=500, edgecolor='red', facecolor='none')
 ```
diff --git a/tutorials/irsa-sia-examples/sia_allwise_atlas.md b/tutorials/irsa-sia-examples/sia_allwise_atlas.md
index 09a331e2..1736d3aa 100644
--- a/tutorials/irsa-sia-examples/sia_allwise_atlas.md
+++ b/tutorials/irsa-sia-examples/sia_allwise_atlas.md
@@ -11,7 +11,7 @@ kernelspec:
   name: python3
 ---
 
-# Image Thumbnails from AllWISE Atlas Images
+# AllWISE Atlas Images
 
 +++
 
@@ -161,7 +161,7 @@ wcs = cutout.wcs
 fig = plt.figure()
 
 ax = fig.add_subplot(1, 1, 1, projection=wcs)
-ax.imshow(cutout.data, cmap='gray_r', origin='lower', 
+ax.imshow(cutout.data, cmap='gray_r', origin='lower',
           vmax = 1000)
 ax.scatter(ra, dec, transform=ax.get_transform('fk5'), s=500, edgecolor='red', facecolor='none')
 ```
diff --git a/tutorials/irsa-sia-examples/sia_cosmos.md b/tutorials/irsa-sia-examples/sia_cosmos.md
index ab452838..81e52082 100644
--- a/tutorials/irsa-sia-examples/sia_cosmos.md
+++ b/tutorials/irsa-sia-examples/sia_cosmos.md
@@ -11,7 +11,7 @@ kernelspec:
   name: python3
 ---
 
-# Image Thumbnails from COSMOS
+# COSMOS
 
 +++
 
diff --git a/tutorials/irsa-sia-examples/siav2_seip.md b/tutorials/irsa-sia-examples/siav2_seip.md
index ddb8ad2f..f1caddcf 100644
--- a/tutorials/irsa-sia-examples/siav2_seip.md
+++ b/tutorials/irsa-sia-examples/siav2_seip.md
@@ -11,7 +11,7 @@ kernelspec:
   name: python3
 ---
 
-# Image Thumbnails from Spitzer Enhanced Imaging Products
+# Spitzer Enhanced Imaging Products
 
 +++
 
diff --git a/tutorials/openuniversesims/openuniverse2024_roman_simulated_timedomainsurvey.md b/tutorials/openuniversesims/openuniverse2024_roman_simulated_timedomainsurvey.md
index 5d8bdb61..41b371d9 100644
--- a/tutorials/openuniversesims/openuniverse2024_roman_simulated_timedomainsurvey.md
+++ b/tutorials/openuniversesims/openuniverse2024_roman_simulated_timedomainsurvey.md
@@ -11,7 +11,7 @@ kernelspec:
   name: python3
 ---
 
-# Explore OpenUniverse 2024 Data Preview: Simulated Roman Time Domain Survey Data
+# Simulated Roman Time Domain Survey Data
 
 +++
 
@@ -26,7 +26,7 @@ By the end of this tutorial, you will:
 
 +++
 
-# Import required modules
+## Install and Import required modules
 
 ```{code-cell} ipython3
 # Install libraries if necessary
diff --git a/tutorials/openuniversesims/openuniverse2024_roman_simulated_wideareasurvey.md b/tutorials/openuniversesims/openuniverse2024_roman_simulated_wideareasurvey.md
index d4b9ca4c..c8d8bfea 100644
--- a/tutorials/openuniversesims/openuniverse2024_roman_simulated_wideareasurvey.md
+++ b/tutorials/openuniversesims/openuniverse2024_roman_simulated_wideareasurvey.md
@@ -11,7 +11,7 @@ kernelspec:
   name: python3
 ---
 
-# Explore OpenUniverse 2024 Data Preview: Simulated Roman Coadds
+# Simulated Roman Coadds
 
 +++
 
@@ -24,6 +24,8 @@ By the end of this tutorial, you will learn how to do the following:
 - Identify a simulated Roman coadd by filter, row, and column.
 - Take a closer look at a simulated coadd file.
 
+## Install and Import required modules
+
 ```{code-cell} ipython3
 # Install libraries if necessary
 !pip install astropy numpy s3fs

From 011684ffc66181869c1bbbfaf1badfba6c0f66a8 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Brigitta=20Sip=C5=91cz?= <bsipocz@gmail.com>
Date: Fri, 19 Jul 2024 18:54:13 -0700
Subject: [PATCH 5/7] CONT: adding latest update from the dev space

---
 .../parquet-catalog-demos/wise-allwise-catalog-demo.md      | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/tutorials/parquet-catalog-demos/wise-allwise-catalog-demo.md b/tutorials/parquet-catalog-demos/wise-allwise-catalog-demo.md
index 6371771d..5479cbb1 100644
--- a/tutorials/parquet-catalog-demos/wise-allwise-catalog-demo.md
+++ b/tutorials/parquet-catalog-demos/wise-allwise-catalog-demo.md
@@ -346,4 +346,8 @@ schema.names
 
 ## About this notebook
 
-This notebook was developed by Troy Raen (raen@ipac.caltech.edu) in conjunction with David Shupe, Jessica Krick and the IRSA Science Platform team at IPAC.
+- Author: Troy Raen (IRSA Developer) in conjunction with David Shupe, Jessica Krick and the IPAC Science Platform team
+- Updated: 2023-7-27
+- Contact: https://irsa.ipac.caltech.edu/docs/help_desk.html
+
++++

From 2e915b102e774da2f4e9f3daaab68f5a647b0380 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Brigitta=20Sip=C5=91cz?= <bsipocz@gmail.com>
Date: Fri, 19 Jul 2024 19:33:49 -0700
Subject: [PATCH 6/7] CI: making test runs a bit more verbose

---
 tox.ini | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tox.ini b/tox.ini
index 437e0f72..927bf520 100644
--- a/tox.ini
+++ b/tox.ini
@@ -35,7 +35,7 @@ commands =
     # Ignore testing the tutorials listed in ignore_testing file
     !buildhtml: bash -c 'find tutorials -name "*.md" | grep -vf ignore_testing | xargs jupytext --to notebook '
 
-    !buildhtml: pytest --nbval-lax --durations=10 tutorials
+    !buildhtml: pytest --nbval-lax -vv --durations=10 tutorials
     buildhtml: sphinx-build -b html . _build/html -D nb_execution_mode=auto -nT --keep-going
 
 pip_pre =

From 1df8ef09584f1d4facb60da6516b8e0898c058e7 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Brigitta=20Sip=C5=91cz?= <bsipocz@gmail.com>
Date: Wed, 24 Jul 2024 20:26:33 -0700
Subject: [PATCH 7/7] Update content requirements file

---
 .../openuniverse2024_roman_simulated_timedomainsurvey.md   | 4 ++--
 .../openuniverse2024_roman_simulated_wideareasurvey.md     | 4 ++--
 tutorials/requirements.txt                                 | 7 ++++---
 3 files changed, 8 insertions(+), 7 deletions(-)

diff --git a/tutorials/openuniversesims/openuniverse2024_roman_simulated_timedomainsurvey.md b/tutorials/openuniversesims/openuniverse2024_roman_simulated_timedomainsurvey.md
index 41b371d9..c272d353 100644
--- a/tutorials/openuniversesims/openuniverse2024_roman_simulated_timedomainsurvey.md
+++ b/tutorials/openuniversesims/openuniverse2024_roman_simulated_timedomainsurvey.md
@@ -29,8 +29,8 @@ By the end of this tutorial, you will:
 ## Install and Import required modules
 
 ```{code-cell} ipython3
-# Install libraries if necessary
-!pip install astropy matplotlib numpy pandas pyarrow s3fs scipy
+# Install libraries if necessary by uncommenting the following
+# !pip install astropy matplotlib numpy pandas pyarrow s3fs scipy
 ```
 
 ```{code-cell} ipython3
diff --git a/tutorials/openuniversesims/openuniverse2024_roman_simulated_wideareasurvey.md b/tutorials/openuniversesims/openuniverse2024_roman_simulated_wideareasurvey.md
index c8d8bfea..c99ec241 100644
--- a/tutorials/openuniversesims/openuniverse2024_roman_simulated_wideareasurvey.md
+++ b/tutorials/openuniversesims/openuniverse2024_roman_simulated_wideareasurvey.md
@@ -27,8 +27,8 @@ By the end of this tutorial, you will learn how to do the following:
 ## Install and Import required modules
 
 ```{code-cell} ipython3
-# Install libraries if necessary
-!pip install astropy numpy s3fs
+# Install libraries if necessary by uncommenting the following
+# !pip install astropy numpy s3fs
 ```
 
 ```{code-cell} ipython3
diff --git a/tutorials/requirements.txt b/tutorials/requirements.txt
index 18f8d69f..14218b6d 100644
--- a/tutorials/requirements.txt
+++ b/tutorials/requirements.txt
@@ -2,12 +2,13 @@
 numpy
 matplotlib
 astropy
-pyvo
+pyvo>=1.5
 scipy
-pyarrow
+pyarrow>=10.0.1
 hpgeom
-pandas
+pandas>=1.5.2
 dask[distributed]
 psutil
+s3fs
 # For supporting myst-based notebooks
 jupytext