Support processing of PDS products with fz file format in Nucleus

[ramesh-maddegoda]

Some of the PDS products processed by Nucleus has compressed version of the data (fitz), and not the uncompressed (fits) in order to save space for archiving the data. Also, in the product label XML, it is referring to the .fits format while the actual file shared with the product label are in .fitz format. This created challenges to determine the reception of a complete product in staging S3 bucket, when the files are uploaded to the staging S3 bucket to be processed by Nucleus.

It is required to,

• Modify the product completion detection algorithm implementation to consider this special case
• Implement a Nucleus component to extract fitz file format to fits file format

References:

• https://heasarc.gsfc.nasa.gov/fitsio/fpack/
• https://heasarc.gsfc.nasa.gov/FTP/software/fitsio/c/docs/fpackguide.pdf

[ramesh-maddegoda]

It was required build a Amazon Linux compatible version of fpack/funpack from the source code

Download the latest 4.3.0 version of CFITSIO
See what's new
Latest V4.3.0 source code package - Compile It Yourself
Unix .tar file cfitsio-4.3.0.tar.gz - see the README file for instructions. The .tar file places unpacks files into a directory named 'cfitsio-4.3.0'.
(The latest version is always available from this link: cfitsio_latest.tar.gz).

The zlib was missing in Amazon Linux EC2 instance that was used to build this. There for it was required to build zlib also.

configure
make -j4
make install

[jordanpadams]

Status: Ramesh working on architecture and implementation in Nucleus

[tloubrieu-jpl]

validate used by Ramesh does not validate the CSS products. Ramesh will upgrade.

[ramesh-maddegoda]

The pull request is available at #78

[ramesh-maddegoda]

The lambda based implementation made for this was working, but it was expensive due to the fact it consumes a large amount time in lambda to download and extract the .fz file. It was decided to come up with a more cost effective option download and extract .fz files.

[tloubrieu-jpl]

@ramesh-maddegoda works on the new more cost-effective design with datasync.

[tloubrieu-jpl]

90% of the development is done.

[jordanpadams]

Status: @ramesh-maddegoda working through some issues with duplicate data in database. Lambda is working as expected. Able to trigger Nucleus workflow. Harvest is failing because our OpenSearch connection is failing for some reason. TBD reason. PR to be submitted before vacation.

[jordanpadams]

Per #80 going to close this out with initial implementation completed. If additional work is required, we will open more tickets.