Stream data location and loading conventions

This page describes conventions for stream data location.

1. Data location

1.1. Directories

Configuration data related to streams is in local directory ./conf.

Parameters that are part of the function parameter structures, stored in directory ./fpsconf/

1.2. Stream location

A stream (= image stream) may reside in the following locations:

• [LOCALMEM] local process memory
• [SHAREMEM] system shared memory .. which may itself be a link to another shared memory
• [CONFFITS] fits file in conf: a file ./conf/shmim.streamname.fits, which may itself be a link to another FITS file
• [CONFNAME] name of fits file configuration: a file ./conf/shmim.streamname.fname.txt contains the name of the disk file to be loaded as the stream, relative to current running directory

Per convention, [LOCALMEM] is downstream and [CONFNAME] is upstream.

2. Loading policy

The default policy is to first look for the stream in LOCALMEM, and proceed upstream until found.

Once source location is found, the downstream locations are updated. For example: search[LOCALMEM]; search[SHAREMEM]; find[CONFFITS]->update[SHAREMEM]->update[LOCALMEM]

3. Implementation

3.1. Scripts (should be in PATH)

milkStreamLink : build sym link between streams

Establishes a stream sym link from TARGET=sourcestream to LINKNAME=streamname. Reads conf/streamlink.streamname.name.txt to identify source stream.

• Create /milk/shm/streamname.im.shm that points to /sourcestream.im.shm
• Runs milk, connects to streamname and writes its size to ./conf/streamlink.streamname.imsize.txt

milkFits2shm : smart loading/updating of FITS to SHM

loading CONF to SHM must use script milkFits2shm

3.2. Stream loading C function (does not check size)

The C function loads stream according to flag settings.

long COREMOD_IOFITS_LoadMemStream(
const char *sname,
uint64_t streamflag,
int *imLOC);

Upon completion, imLOC can have one of five values:

#define STREAM_LOAD_SOURCE_FAILURE   0
#define STREAM_LOAD_SOURCE_LOCALMEM  1
#define STREAM_LOAD_SOURCE_SHAREMEM  2
#define STREAM_LOAD_SOURCE_CONFFITS  3
#define STREAM_LOAD_SOURCE_CONFNAME  4

3.3. FITS loading C functions (checks size)

Two C functions provided to load FITS files from disk to shared memory, with size testing:

long AOloopControl_IOtools_2Dloadcreate_shmim(
const char *name,     // stream name
const char *fname,    // file name
long xsize,           // X size
long ysize,           // Y size
float DefaultValue);

long AOloopControl_IOtools_3Dloadcreate_shmim(
const char *name,
const char *fname,
long xsize,
long ysize,
long zsize,
float DefaultValue);

The logic implemented is :

// Is present in local mem ?
// YES: goto (LOADFITS) 
// NO : Can read shared memory ?
//      YES: SHM size OK ?
//           YES: Load shared memory
//                go to (LOADFITS)
//           NO : Delete SHM, create new one with correct size [STATUS = 0]
//                goto (LOADFITS)
//      NO : Create new SHM with correct size [STATUS = 1]
//           goto (LOADFITS)
//
// (LOADFITS)
// Can load FITS ?
// YES: FITS size OK ?
//      YES: Load FITS and write to memory [STATUS = 2] 
//      NO : FITS image has wrong size-> do nothing [STATUS = 3]
// NO : exit, [STATUS = 4 or 5]