Skip to content

Latest commit

 

History

History
136 lines (91 loc) · 3.17 KB

cloud-storage.rst

File metadata and controls

136 lines (91 loc) · 3.17 KB
Raw

Cloud Storage Tutorial

Table of Contents

This tutorial covers access via two methods: pittgoogle-client (with some direct use of the Google Cloud API), and the gsutil CLI.

Prerequisites

  1. Complete the initial setup. In particular, be sure to:
    • install and/or Install the command-line tools <install gcp cli>.
    • service account
    • Set your environment variables <set env vars>

Python

Setup

Imports

import fastavro
from google.cloud import storage
from matplotlib import pyplot as plt
import os
from pathlib import Path
import pittgoogle

Name some things

# fill in the path to the local directory to which you want to download files
local_dir = ''

my_project_id = os.getenv('GOOGLE_CLOUD_PROJECT')
pgb_project_id = pittgoogle.utils.ProjectIds.pittgoogle

Download files

Download alerts for a given objectId

objectId = 'ZTF17aaackje'
bucket_name = f'{pgb_project_id}-ztf-alert_avros'

# Create a client and request a list of files
storage_client = storage.Client(my_project_id)
bucket = storage_client.get_bucket(bucket_name)
blobs = bucket.list_blobs(prefix=objectId)

# download the files
for blob in blobs:
    local_path = f'{local_dir}/{blob.name}'
    blob.download_to_filename(local_path)
    print(f'Downloaded {local_path}')

Plot cutouts and lightcurves

The functions in this section were adapted from https://github.com/ZwickyTransientFacility/ztf-avro-alert/blob/master/notebooks/Filtering\_alerts.ipynb.

Open a file (see the previous section to download files)

paths = Path(local_dir).glob('*.avro')
for path in paths:
    with open(path, 'rb') as fin:
        alert_list = [r for r in fastavro.reader(fin)]
    break
alert_dict = alert_list[0]  # extract the single alert packet

print(alert_dict.keys())

Plot cutouts

pittgoogle.figures.plot_cutouts(alert_dict)
plt.show(block=False)

Cast to a dataframe and plot lightcurves

lc_df = pittgoogle.utils.Cast.alert_dict_to_dataframe(alert_dict)
pittgoogle.figures.plot_lightcurve(lc_df)
plt.show(block=False)

Plot everything together

pittgoogle.figures.plot_lightcurve_cutouts(alert_dict)
plt.show(block=False)

Command line

See also:

Get help

gsutil help
gsutil help cp

Download a single file

# fill in the path to the local directory to which you want to download files
local_dir=
# fill in the name of the file you want. see above for the syntax
file_name=
# file_name=ZTF17aaackje.1563161493315010012.ztf_20210413_programid1.avro
avro_bucket="${pgb_project_id}-ztf-alert_avros"

gsutil cp "gs://${avro_bucket}/${file_name}" ${local_dir}/.