"det_flux" error

[redeaglekr]

Hi all,The problem is related to depreciated label ("kspsap_flux") for the QLP de-trended flux.

I tried to download all available the light curves for KIC 12690461, keeping the "kspsap_flux" as the required flux column. But found that such a column is not there in the headers. I found that this is due to change of label from "kspsap_flux" (untill sector 55) to "det_flux" (from sector 56).

Example

import lightkurve
sr = lk.search_result("KIC xxxxx", sector = np.arange(1, 56, 1, dtype = int), author = ["QLP", "TESS"], mission = "TESS")
lcc = sr.download_all()
lcc = lcc.stitch(corrector_func = lambda x: x.remove_outliers(sigma=5).normalize(unit = "unscaled"))
print(lcc.keys())
lcc = lcc.select_flux("kspsap_flux")

##This issue is not there when I download data upto sector 55 of rthe same stars, as it will show "kspsap_flux"

sr = lk.search_result("KIC xxxxx", sector = np.arange(1, 55, 1, dtype = int), author = ["QLP", "TESS"], mission = "TESS")
lcc = sr.download_all()
lcc = lcc.stitch(corrector_func = lambda x: x.remove_outliers(sigma=5).normalize(unit = "unscaled"))
print(lcc.keys())
lcc = lcc.select_flux("kspsap_flux")

###or when I try to download sector 56 only,

sr = lk.search_result("KIC xxxxx", sector = 56, author = ["QLP", "TESS"], mission = "TESS")
lcc = sr.download_all()
lcc = lcc.stitch(corrector_func = lambda x: x.remove_outliers(sigma=5).normalize(unit = "unscaled"))
print(lcc.keys())
lcc = lcc.select_flux("det_flux")

#### Environment

-  platform (e.g. Linux, OSX, Windows):
-  lightkurve version (e.g. 1.0b6):
-  installation method (e.g. pip, conda, source):

[github-actions]

Hi there! 👋 Thank you for opening your first Lightkurve issue! 🙏 One of our maintainers will get back to you as soon as possible. 👩‍🚀 You can expect a response within 7 days. 📅 If you haven’t heard anything by then, feel free to ping this thread. 🛎️ We love that you are using Lightkurve and appreciate your feedback. 👍

[tylerapritchard]

Hi Sree, thanks for reaching out! This sounds like a pipeline change, we'll look into it.

Currently the QLP lightcurve reader uses SAP flux as the default, which is why we can read in the files. I’m sure you’ve already tried this, but in the meantime you should be able to create a light curve collection with the flux set to kspasp for sectors 1-55 and det_flux in sector 56 if you so choose by doing something like this:

sr55 = lk.search_lightcurve("KIC 12690461", sector=np.arange(56), author = ["QLP", "TESS"], mission = "TESS")
sr56 = lk.search_lightcurve("KIC 12690461", sector = 56, author = ["QLP", "TESS"], mission = "TESS")

lcc = sr55.download_all()
lcc_56 = sr56.download()
for i in np.arange(len(lcc)):
    lcc[i] = lcc[i].select_flux("kspsap_flux")
lcc_56 = lcc_56.select_flux("det_flux")

lcc.append(lcc_56)

[orionlee]

To access QLP detrended lightcurves across sectors 1-55 and 56+, I'm afraid one needs
to switch between det_flux and kspsap_flux.

One possibility that lightkurve could make users' life simpler it to extend select_flux() to accept an ordered list of columns as the parameter, something like.

# use det_flux. It the column does not exist, use kspsap_flux
lc.select_flux(["det_flux", "kspsap_flux"])

For reference, changes in QLP starting from sector 56:
#1389 (comment)

[Nschanche]

Thanks for referencing the other issue @orionlee, I had missed that before. The QLP reader currently does use different flags before/after sector 56 for flux_err, but it uses sap_flux as flux by default. We can update this to use kspsap_flux or det_flux depending on sector.
The stitching process appears to remove columns that do not match, so there is no longer det_flux or kspsap_flux in the column list. The stitched columns in this example are only include ['time','flux','flux_err','cadenceno','sap_flux','quality','orbitid','sap_x','sap_y','sap_bkg','sap_bkg_err'], so adding a list option in select_flux still not be able to find the desired columns.
As it sounds like the change is in name only from Glen's comment, I think the way forward would be to modify the QLP reader to make column names consistent. I will take a look at making this change.

[orionlee]

I recalled there was some reason sap_flux instead of detrended flux was deliberately used as the default for QLP, but I do not remember the specifics.

Changing the default to detrended flux will be a breaking change.

---

The name change is because QLP team changed the detrending algorithm. From Glen Petitpas when I first asked him about the changes:

We stopped using KSP for detrending so we have renamed some things to make the naming more generic and future-proof. ...

---

Considerations if we change QLP reader to make column names consistent.

1. it could be a breaking change, e.g., if the detrended column names are renamed to det_ variant, it could breaking existing codes out there (that work on pre-sector 56 QLP data), assuming it is ksp variant.
2. I wonder if there would be cases the difference in detrending algorithm might be important.

One possibility is to make the renaming an option that users need to explicitly opt-in, e.g., add a new rename_ksp_to_det parameter to SearchResult.download() / QLP reader:

lcc = sr.download_all(rename_ksp_to_det=True)
lc = lcc.stitch()
lc = lc.select_flux("det_flux")