Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Should be ReducedFit.mdata a cached or lazy property? #33

Open
juanep97 opened this issue Oct 26, 2023 · 1 comment
Open

Should be ReducedFit.mdata a cached or lazy property? #33

juanep97 opened this issue Oct 26, 2023 · 1 comment
Labels
help wanted Extra attention is needed

Comments

@juanep97
Copy link
Owner

juanep97 commented Oct 26, 2023

Right now, RawFit.mdata, ReducedFit.mdata are properties that read and return the data from the files every time they are used. There could be two alternatives with different advantages and drawbacks:

  • Make them cached properties so that they read the file only one, saving quite a bit of disk read access. This would make some functions and views slightly faster.
  • Make RawFit.mdata a cached property that reads data only one. Make ReducedFit.mdata a cached property that computes the value on the fly, applying the master calibrations frames on the fly to the rawfit. This would make it the pipeline processes only slightly slower, however, for the iop4admin views using the data, such as the stats on the detailed views, it might add some seconds to the page load. Some tests are needed to check if these differences are significant (applying the calibration frames, once they are built, is quite fast).

The main advantage of this point is that disk space would be reduced by a factor of x5. Right now, since the reduced FITS are saved using float64 to keep precision (#30 ), they take x4 times the size of the raw images (uint16 usually). This method would completely eliminate these files from the disk. the .build_file would only build and save the WCS header to disk (a few kilobytes at most), and the .header would read this.

@juanep97 juanep97 added the help wanted Extra attention is needed label Oct 26, 2023
@juanep97
Copy link
Owner Author

juanep97 commented Oct 26, 2023

Alternatively, we could make ReducedFit.mdata return the files from the disk if they exist, otherwise just build it, save it, and return it. Then we could delete old reduced fits file and only keep the last months, that are the ones more likely to be consulted in the admin, freeing quite a lot of space too. This however will mean more writes, which is what wores out the disk more (also not very important, but still).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

1 participant