Skip to content

HATSpy: User Manual ‐‐ V0.01

Jump to bottom
Guillermo Giménez de Castro edited this page Jul 27, 2025 · 42 revisions

Guigue (a.k.a. Guillermo Giménez de Castro)

guigue@craam.mackenzie.br

Copyright © 2025 by Guigue

All rights reserved

Introduction

HATSpy is a set of programs written in python and C used to reduce and analyze data from the HATS telescope organized as an ipython profile. This manual was written and tested in the following OS: Linux Debian 11 with kernel 5.10.0-35-amd64, python 3.9 and ipython3 7.20; and Ubuntu 20.04.6, kernel 5.4.0-216-generic, python 3.8.10 and ipython3 7.13.

Installation

HATSpy is developed in python 3.7 or later and standard C. The most updated version of the software can be downloaded from https://github.com/guigue/CRAAM-Instruments/HATS. The software can be installed in any directory, and environment variables must be defined.

  1. PYTHONPATH: must include the path where the python code is installed.
  2. HATSXMLPATH: must point to the directory where .xml HATS files are copied.
  3. HATS_DATA_InputPath: Directory where data is downloaded. We will talk about the structure of this directory later.
  4. HATS_WS_InputPath: weather station data directory.
  5. HATS_FFTProgram: full path to the binary code of the FFT program.

hatspy profile

To create this profile you run the command ipython3 profile create hats. The go to the directory $HOME/ipython/profile_hats/startup and create the file 00-hats.py with the following lines

              import HATS 
              from HATSTools import *
              import numpy as np
              from matplotlib.pyplot import plot
              from matplotlib import pyplot as plt
              from astropy import units as u
              from astropy import constants as c
              from astropy.time import Time
              plt.ion()

Finally, create an alias:

(BASH) > alias hatspy='ipython3 --pylab --profile=hats'

(CSH) > alias hatspy 'ipython3 --pylab --profile=hats'

You can (have to) add one of the previous lines in $HOME/.bashrc or $HOME/.cshrc files. Next shell you open, you should be able to run

> hatspy
Python 3.8.10 (default, Mar 18 2025, 20:04:55) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.13.0 -- An enhanced Interactive Python. Type '?' for help.

IPython profile: hats
Using matplotlib backend: TkAgg

HOMS Configuration

The HATS Operation, Monitoring and Storage (HOMS) computer at the OAFA observatory is configured as follows:

                        PYTHONPATH=/opt/HAX/python
                        HATSXMLPATH=/opt/HAX/python/XMLTables
                        HATS_DATA_InputPath=/homs/HATS/data
                        HATS_WS_InputPath=/homs/HATS/data/aux
                        HATS_FFTProgram=/opt/HAX/python/HATS_fft

HATS Data Model

HATS data is distributed between binary and ascii files. Astronomical data, or just data, is stored in .rbd binary files (for Raw Binary Data), auxiliary data is stored in .aux binary files. Auxiliary files are stored in a subdirectory of the main data directory, also the weather station data is stored in the AUX subdirectory. The log files are stored in another subdirectory at the same data directory level. Graphically:

       root
         |
         |__data
         |    | 
         |    |__ aux 
         |
         |__log

The data model is described in .xml files located in the HATSXMLPATH directory defined during the installation process in an environment variable (see above). .rbd are defined in HATSDataFormat.xml while .aux files in HATSAuxFormat.xml. Filename format is defined in the acquisition software (HOMS_data_transfer_daemon.c). The name is as follows

  RBD files: hats-YYYY-MM-DDTHH00.rbd
  AUX files: hats-YYYY-MM-DDTHH00.aux

where YYYY-MM-DDTHH00 is the observation time in ISO format (see that minutes are set to 00, even if the observation starts after 0 minutes). Every hour a new pair of RBD/AUX files is created.

RBD are fixed-length binary files time-stamped 1-millisecond Golay and temperature sensors readouts in ADC units (integer numbers from $0$ to $10^{24}-1$), and the time-stamp in husec (100 µs) since 0 UT of the day among other data. AUX files are time-stamped, fixed-length binary 1-second files with right ascension, declination, azimuth, and elevation read from the telescope mount; flags and special codes for the operation modes.

Weather Station

The weather station data is stored in the AUX directory with the name hats-YYYY-MM-DD.ws. They are ascii files with time-stamped records. hatspy has a special routine to read these data files.

Reading Data

The command

In [1] h = HATS.hats(‘2025-04-08 1200’)

will read the files hats-2025-04-08T1200.rbd and hats-2025-04-08T1200.aux and create an object called h. This object is formed by two objects h.rbd and h.aux.

RBD Object

The RBD object has three ndarrays: rData , cData and Deconv. The first one is an image of the sent data from HICS to HOMS, i.e. uncalibrated data in ADC units, and the time in husec. The second array has the mVolts-converted data only, i.e. it has not husec. Finally, Deconv is an array with the time in python datetime format, the extracted amplitude of the Golay signal in mVolts, and the time in husec (this is needed since in the deconvolution process we change the time resolution). Below we show the description of each array.

rData

Name dtype Description
sample <u4 32b unsigned integer with a sequential number.
sec <u4 32b unsigned integer with the seconds since 0 UT.
ms <u2 32 unsigned integer with the milliseconds since 0 UT.
husec <u8 64 unsigned integer with the time in 100 µm.
golay <i4 32b integer with the Golay output in ADC units.
chopper <i4 32b integer with the chopper voltage in ADC units.
temp_hics <i4 32b integer with the temperature of the embedded computer HICS in ADC units
temp_env <i4 32b integer with the temperature of the external case of the Golay cell in ADC units
temp_golay <i4 32b integer with the temperature of the Golay cell in ADC units

cData

Name dtype Description
golay <f8 64b floating with the Golay output in mVolts.
chopper <f8 64b floating with the chopper mVolts.
temp_hics <f8 64b floating with the embedded computer HICS temperature in °C
temp_env <f8 64b floating with the environment temperature in °C
temp_golay <f8 64b floating with the Golay temperature in °C

Deconv

Name dtype Description
time O a datetime.datetime object with the current time.
husec <u8 a 64b unsigned integer with the mean time of the FFT in 100 µm. We take 128 sequential values to compute the FFT. husec corresponds to the center of this time
amplitude <f8 a 64b floating with the amplitude of the Golay signal in mVolts.

AUX Object

The AUX object has one ndarray called Data with the following fields (dtypes more rigorously)

Name dtype Description
husec <u8 64b unsigned integer with the time in 100 µm
jd <f8 64b floating with the julian day
sid <f8 64b floating
elevation <f8 64b floating with the current elevation in degrees
azimuth <f8 64b floating with the current azimuth in degrees
right_ascension <f8 64b floating with the current right ascension in degrees
declination <f8 64b floating with the current declination in degrees
ra_rate <f8 64b floating with the current right ascension rate (speed) in degrees/s
dec_rate <f8 64b floating with the current declination rate (speed) in degrees/s
object <i4 32b integer with the object code
opmode <i4 32b integer with the operation mode code
time O datetime.datetimeobject with the time

Weather Station Data

The command

ws = HATS.ws('YYYY-MM-DD')

reads the weather data and stores it in the object called ws. The argument of the command is the time in ISO format. Files have the data for one day long in 5-second time bins. The ws object has a dictionary called data with the following keys

Name type Description
time O a datetime.datetime object with the time
temperature float64 a 64b floating with the air temperature in °C
humidity float64 a 64b floating with the air humidity in %
pressure float64 a 64b floating with the air pressure in HPa

There is a pwv key which is empty. This is an error that future HATSpy versions will correct

HATS Methods

self.getTimeAxis()

Converts husec values into datetime.datetime objects. It takes two arguments: the husec field and MetaData and returns a ndarray of datetime objects with size h.rbd.rData['husec'].shape[0].

In [2]: t=h.getTimeAxis(h.rbd.rData['husec'],h.MetaData)

self.extract()

Extracts a data interval from self. It takes an argument as a list of 2 elements of type datetime.datetime

hh=h.extract([Time('YYYY-MM-DD HH:mm:ss').datetime,Time('YYYY-MM-DD HH:mm:ss').datetime])

The return hh is an object of the HATS class.

Extract, save and recover

It is very practical for future uses to save the extracted object in a binary file. python has the pickle binary format that can be used. Then the previous instruction can be changed to

hh=h.extract([Time('YYYY-MM-DD HH:mm:ss').datetime,Time('YYYY-MM-DD HH:mm:ss').datetime], save=True, pklname='My_Name.pkl')

This command will create a new object hh and save its contents to the file My_Name.pkl . Later this file can be read with

mh = HATS.hats(pklname='My_Name.pkl')

Follows a real example:

In [1]: h=HATS.hats('2025-07-20 1400')
In [2]: hh=h.extract([Time('2025-07-20 14:00').datetime,Time('2025-07-20 14:04').datetime],save=True,pklname='h_2025_07_20-14_00-14_04.pkl')
In [3]: scans = HATS.hats(pklname='h_2025_07_20-14_00-14_04.pkl')

The new created object scans has HATS data for the interval '2025-07-20 14:00' to '2025-07-20 14:04'.

Concatenating 2 files (or the addition overload)

Two objects of the HATS class can be merged into a single one by just using the + symbol

In [3]: h = h1 + h2

with h1and h2 two HATS objects.

self.plot()

It is a simple plotting method; it opens a Figure window and returns the time in an array of datetime.datetime objects

In [4]: t=h.plot()

The method plots self.rbd.Deconv['amplitude'] in function of time.

self.smooth()

It computes the running mean of the Deconv['amplitude'] array. It takes one argument with the number of data bins that will be used to compute the smooth

In [5]: h.smooth(N)

where N is a positive integer value. There is no output return. The method adds a new column in the Deconv array self.rbd.Deconv.smooth : a 64b floating with the smoothed amplitudes. Since the original data (amplitude) is preserved, the method can be run any number of times as needed until satisfied.

Weather Station Methods

self.pwv()

It takes no mandatory argument and computes the Precipitable Water Vapor content (PWV). The result is stored in the PWV dictionary field.

In [6]: ws.pwv()

The PWV obtained is derived from simple formulas (Butler, B., ALMA memo 237, 1998). The H2O scale can be changed with the optional input parameter h2o_scale=ff.ff where ff.ff is the scale height in km. Default is 2.0 (km)

self.plot()

It takes an optional input parameter var=str, where str can be either 'temperature', 'pressure', 'humidity', or (if it is already calculated) 'PWV'; the default is 'temperature' when no parameter is given.

In [7]: ws.plot(var=`pressure`)

to plot the pressure in %.

Clone this wiki locally