Retrieving HST ACS FLT and RAW files

[Borlaff]

Hi,

I am trying to find and download all the FLT and RAW files of Hubble Space Telescope observations using astroquery.mast. The calibration level for these files is 0 (RAW) -- 2 (FLT). However, when I use astroquery I can only find the calibration level 2 files for some instruments (WFPC/WFC, NICMOS, WFC3) not ACS. Here I attach a minimal reproducible example.

import numpy as np

from astroquery.mast import Observations
c = coordinates.SkyCoord(186.75795, +33.55426, frame='icrs', unit="deg")

obs_table = Observations.query_criteria(coordinates=c, radius=0.5,
                                        calib_level=2, obs_collection="HST")
set(np.array(obs_table["instrument_name"]))

Output:

{'COS/NUV',
'FOS/BL',
'FOS/RD',
'NICMOS/NIC2',
'NICMOS/NIC3',
'STIS/CCD',
'WFC3/IR',
'WFC3/UVIS',
'WFPC/PC',
'WFPC/WFC',
'WFPC2',
'WFPC2/PC',
'WFPC2/WFC'}

However, I know from MAST webpage that there are ACS/WFC observations in that region. What I am missing? Thank you
Alex

---

[bsipocz]

cc @jaymedina

[jaymedina]

Hi @Borlaff

From the link you shared, the ACS products seem to be level 3 and above. Running a call for calib_level 2 and 3 and retrieving the instrument_names of the products show me there are some ACS products in there:

>>> table = Observations.query_criteria(coordinates=c, radius=0.5, calib_level=[2, 3], obs_collection='HST')
>>> set(np.array(table['instrument_name']))
{'WFPC2/PC', 'COS/FUV', 'STIS/CCD', 'NICMOS/NIC3', 'FOS/BL', 'NICMOS/NIC1', 'COS/NUV', 'WFPC/WFC', 'NICMOS/NIC2', 'ACS/SBC', 'STIS/FUV-MAMA', 'ACS/WFC', 'WFPC2', 'WFPC2/WFC', 'FOS/RD', 'WFC3/IR', 'ACS/HRC', 'WFC3/UVIS', 'WFPC/PC', 'STIS/NUV-MAMA'}

[Borlaff]

Thank you so much! I tried that, but my interest is in downloading the lower level products. What is the correct procedure to download RAW and FLT/FLC files from HST / ACS using astroquery?

Alex

[jaymedina]

We're investigating this now and will get back to you soon

[jaymedina]

The calib_level column in your output obs_table will return the highest level product for that observation I believe. To retrieve a specific calibration level, you need to take one extra step and make a call to get_products_list which should return all the products corresponding to each observation. So your call would look like this:

from astropy import coordinates
from astroquery.mast import Observations
import numpy as np

c = coordinates.SkyCoord(186.75795, +33.55426, frame='icrs', unit="deg")
obs_table = Observations.query_criteria(coordinates=c, radius=0.5, obs_collection="HST")
products_table = Observations.get_product_list(obs_table)

Now you have the true set of products, from levels 0-4, for each observation. From here you can filter down for only FLC/FLT and RAWs using filter_products and the extension criteria. The reason I'm using extension instead of calib_level is because ever since the HAP release, there are some HAP SVM drizzle products that are labeled calib_level=2:

filtered_products = Observations.filter_products(products_table, extension=['flt.fits', 'flc.fits', 'raw.fits')

Unfortunately the product outputs do not have the instrument_name column, but the ACS products should be identifiable via the productFilename column:

for row in filtered_products:
    fn = row['productFilename']
    if 'acs' in fn:
        print(fn)

Here are the first 10 rows I get of this output. For the ACS products there are FLCs and FLTs but no RAWs.

hst_10493_11_acs_wfc_f814w_j9dd11xg_flc.fits
hst_10493_11_acs_wfc_f814w_j9dd11xg_flc.fits
hst_10493_11_acs_wfc_f814w_j9dd11xg_flc.fits
hst_10493_11_acs_wfc_f814w_j9dd11xg_flc.fits
hst_10493_11_acs_wfc_f814w_j9dd11xh_flc.fits
hst_10493_11_acs_wfc_f814w_j9dd11xh_flc.fits
hst_10493_11_acs_wfc_f814w_j9dd11xh_flc.fits
hst_10493_11_acs_wfc_f814w_j9dd11xh_flc.fits
hst_10493_11_acs_wfc_f814w_j9dd11xj_flc.fits
hst_10493_11_acs_wfc_f814w_j9dd11xj_flc.fits

---

TLDR: Use the extension filter after the product-retrieval step (filter_products) rather than the calib_level filter in the observations-retrieval step (query_criteria).

Hope this helps! We are working on making astroquery.mast more intuitive and user-friendly and these user questions give us ideas on what to prioritize. I'm seeing that the calib_level filter is not super intuitive and the observations --> products process needs to be better documented.

[Borlaff]

Thank you so much for your answer. I tested it the code above and it works, but it does however miss some observations. Some HST ACS observations do not have the string "acs" inside their product filename. For example, from the same query above:

from astropy import coordinates
from astroquery.mast import Observations
import numpy as np

c = coordinates.SkyCoord(186.75795, +33.55426, frame='icrs', unit="deg")
obs_table = Observations.query_criteria(coordinates=c, radius=0.5, obs_collection="HST")
products_table = Observations.get_product_list(obs_table)
filtered_products = Observations.filter_products(products_table, extension=['flt.fits'])
filtered_products[1500:1504]

obsID	obs_collection	dataproduct_type	obs_id	description	type	dataURI	productType	productGroupDescription	productSubGroupDescription	productDocumentationURL	project	prvversion	proposal_id	productFilename	size	parent_obsid	dataRights	calib_level
23962280	HST	image	j6fy03jhq	DADS FLT file - Calibrated exposure ACS/WFC3/STIS/COS	S	mast:HST/product/j6fy03jhq_flt.fits	SCIENCE	--	FLT	--	CALACS	10.3.5 (08-Feb-2022)	9033	j6fy03jhq_flt.fits	168442560	26314105	PUBLIC	2
23962280	HST	image	j6fy03jhq	DADS FLT file - Calibrated exposure ACS/WFC3/STIS/COS	S	mast:HST/product/j6fy03jhq_flt.fits	SCIENCE	--	FLT	--	CALACS	10.3.5 (08-Feb-2022)	9033	j6fy03jhq_flt.fits	168442560	26314119	PUBLIC	2
23962281	HST	image	j6fy03jiq	DADS FLT file - Calibrated exposure ACS/WFC3/STIS/COS	S	mast:HST/product/j6fy03jiq_flt.fits	SCIENCE	--	FLT	--	CALACS	10.3.3 (24-May-2021)	9033	j6fy03jiq_flt.fits	10607040	24816022	PUBLIC	2
23962281	HST	image	j6fy03jiq	DADS FLT file - Calibrated exposure ACS/WFC3/STIS/COS	S	mast:HST/product/j6fy03jiq_flt.fits	SCIENCE	--	FLT	--	CALACS	10.3.3 (24-May-2021)	9033	j6fy03jiq_flt.fits	10607040	26314110	PUBLIC	2

For the flt/flc fits somehow we would need a feature that allows to trace back the observations that contain "acs" in the productFilename OR that are jXXXXXXXX_flt.fits / flc.fits. The images with the long productFilename are associated to the project HAP and the ones with the shorter jX style have CALACS in the project column. While we can keep both in the query, there is no immediate way to tell if there are duplicates between those formats.

[jaymedina]

I will bring this up to the database management team. Perhaps it's possible to add instrument_name as a product metadata as well as an observations metadata so that all products for a given instrument can be listed without any implicit for-loop code, and therefore making duplicates easier to catch. Thanks for the note!