Repository navigation
Replies: 5 comments 1 reply
|
@EranOfek , @yossishv , @chentishler: although the PROC images themselves are not saved, we need this table in order to filter the src_proc table by observation time, mounts/telescopes, observation conditions, image quality, etc. (via JOINs). |
|
@EranOfek, a first test of this table is ready. The table was created from the column choices in this post, and the epoch headers of the same 3 visits as in #1409 (1,440 epoch sub-images) were loaded into The two new functions are pushed to |
|
@chentishler, as in #1409, I added a draft of the |
|
It will be good to have a single table with all the details. May I suggest to do it in the wiki (like we have for the headers/catalosgs; so it want get lost in the endless discussions) and to refer to the discussions here... Next, I am not sure I am reading this correctly, but I can't find the index description for all the columns. |
|
@chentishler The script that creates |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
We want a new ClickHouse table,
image_proc, that keeps the image header of every LAST v1 epoch image (one row per epoch sub-image, i.e. about 480 rows per visit: 20 epochs × 24 sub-images). The header is taken from the visit fileproc_Head_1.mat, which holds all 20 × 24 epoch headers. Since #1417 the pipeline writes the epoch catalogs into MAT files instead of the 480proc_CatFITS files (productionlv1.1.2, e.g. the night 2026-10-20 has none). For older visits the header is taken from the third layer (HDU 3) of theirproc_Catfiles; the content is the same. Below is a first proposal in the same format as #1409 (image_visit): for every header keyword (except HISTORY and COMMENT), whether to keep it as a column, its type, default value, the storage resolution it needs, and how many distinct values to expect. Please comment on the ✅/❌ choices.Related: #1409 proposes the coadd image table
image_visitand #1411 the source tablessrc_visitandsrc_procin the same format. Once this proposal is settled, it will be used to create the actual ClickHouse tableimage_proc.What the proposal is based on
LAST.01.02.01_20260405.165513.627_clear_1351_001_001_001_sci_proc_Cat_1.fits, HDU 3 (pipelinelv1.0.5).proc_Catfiles of the 26 evening visits (16:55–22:15 UT) of the night 2026-04-05 on cameras LAST.01.02.01 and LAST.01.03.01: 509 epochs. These are the same files as the src_visit and src_proc: ClickHouse tables for LAST v1 source catalogues #1411 sample.image_visitin visit_images: a ClickHouse table for LAST v1 coadd headers #1409. They are a starting point: which keywords are really needed for epochs is still to be decided.lv1.1.2(night 2026-10-20 on euclid) have the same four new keywords as the coadd header (visit_images: a ClickHouse table for LAST v1 coadd headers #1409):SCHEDULR(one value per visit) andQUICFWHM(the camera software's quick FWHM estimate of each raw image, in pixels, so one value per epoch) from the raw image header, andPT_FNORM,PT_NORMUfrom the absolute calibration (PhotCalibTrans: drop the geometric collecting area #1431). Like the otherPT_values,PT_FNORMof an epoch is the coadd's value, whilePT_1_V1is the epoch's own, so in an epoch rowPT_FNORMis notPT_1_V1divided by the telescope area. Older products do not have the four keywords and get the defaults.MAXBCKDand the four keywords above were added). The epoch header has no coadd-only keywords (NCOADD,AVNCOADD,MINCOADD,COADDOP,SUBLEVEL,MINJD,MAXJD,MAXBCKD,ID_COADD,ID_PROCF,ID_PROCL), and also lacksSPATQ50…SPATQ95,SPATNBIN,RP_MRMS,RP_MMRMS,GM_RATEX,GM_RATEY,GM_STDX,GM_STDY,SNCHI2andOPT_CLN. It has three extra keywords of the FITS extension layer (XTENSION,PCOUNT,GCOUNT).coadd_Catfiles). "✅ in N/12216" means equal except in rows where the epoch or the coadd header lacks the keyword. This column is meant to help decide which columnsimage_procneeds, since the coadd values can also be taken fromimage_visitthroughid_coadd.Design-Database-Pipeline-ClickHouse.xlsx, tab Images,proc_imagescolumn) already has this column, and with which name and type.-replaced by_. IDs and theUPIX_*values areUInt64; each image-section keyword (*SEC,OVERSCAN) becomes fourInt16columns<keyword>_xmin,_xmax,_ymin,_ymax.image_visit. With about 323 columns (297 keywords, the image sections taking four columns each, plusingestion_time_jdandid_coadd) that is some tens of GB per year after compression.Sorting and indexes
Proposed, as for
image_visit: the table is sorted byid_proc(ORDER BY id_proc). The observation time sits in the highest bits of the ID, so this order is close to time order. Additional indexes will be defined by @chentishler.Keyword table
Click a group to open it.
FITS structure (0 of 11 kept)
XTENSIONLowCardinality(String)PCOUNTUInt8GCOUNTUInt8BITPIXInt8NAXISUInt8NAXIS1UInt16int16NAXIS2UInt16int16EXTENDUInt8BZEROFloat32BSCALEFloat32LONGSTRNLowCardinality(String)Identification, files and versions (17 of 28 kept)
FILENAMEStringPROJNAMELowCardinality(String)FULLPROJLowCardinality(String)NODENUMBUInt8int16MOUNTNUMUInt8int8CAMNUMUInt8int8CROPIDUInt8int16CCDIDUInt8int16LEVELLowCardinality(String)IMTYPELowCardinality(String)VERSIONUInt8SUBDIRStringchar16DIRYEARUInt16int16DIRMONUInt8int8DIRDAYUInt8int8FILETIMEStringstringTIMEStringPIPEVERLowCardinality(String)char32GITASTRPLowCardinality(String)GIT_CAMRLowCardinality(String)GITFOCUSLowCardinality(String)ID_PROCUInt64int64ID_RAWUInt64int64BIAS_IMLowCardinality(String)ID_DARKUInt64int64FLAT_IMLowCardinality(String)ID_FLATUInt64int64PSTATUSUInt32Target and observation time (14 of 14 kept)
OBJECTLowCardinality(String)char16FIELDIDLowCardinality(String)char16SCHEDULRLowCardinality(String)DATE-OBSDateTime64(3,'UTC')DateTime64(3)JDFloat64float64JD_STARTFloat64MIDJDFloat64float64LSTFloat32float64CRDATEDateTime64(3,'UTC')EXPTIMEFloat32float32EXPMODELowCardinality(String)COUNTERUInt32int16FILTERLowCardinality(String)char8AIRMASSFloat32float32Observatory and mount (21 of 23 kept)
TIMEZONEFloat32OBSLONFloat64float32OBSLATFloat64float32OBSALTFloat32float32EQUINOXFloat32M_RAFloat32float32M_HAFloat32float32M_DECFloat32float32M_AZFloat32M_ALTFloat32M_JRAFloat32float32M_JDECFloat32float32M_ARAFloat32float32M_AHAFloat32float32M_ADECFloat32float32M_ADRAFloat32float32M_ADHAFloat32float32M_ADDECFloat32float32TRK_RAFloat32float32TRK_DECFloat32float32MNTTEMPFloat32float32FOCUSFloat32Int32PRVFOCUSFloat32Int32Camera and detector (22 of 22 kept)
CAMNAMELowCardinality(String)CAMTEMPFloat32CAMCOOLFloat32float32CAMMODEUInt8CAMGAINUInt16float32CAMOFFSUInt16float32GAINFloat32ORIGGAINFloat32float32READNOIFloat32float32DARKCURFloat32float32SATURVALFloat32float32NONLINFloat32float32BINXUInt8float32BINYUInt8float32LIGHTSECInt16:lightsec_xmin,_xmax,_ymin,_ymaxOVERSCANInt16:overscan_xmin,_xmax,_ymin,_ymaxCCDSECInt16:ccdsec_xmin,_xmax,_ymin,_ymaxORIGSECInt16:origsec_xmin,_xmax,_ymin,_ymaxint16ORIGUSECInt16:origusec_xmin,_xmax,_ymin,_ymaxint16UNIQSECInt16:uniqsec_xmin,_xmax,_ymin,_ymaxint16EXCLSECInt16:exclsec_xmin,_xmax,_ymin,_ymaxORIGESECInt16:origesec_xmin,_xmax,_ymin,_ymaxBackground and image statistics (13 of 13 kept)
MEANBCKFloat32float32MEDBCKFloat32float32STDBCKFloat32float32MINBCKFloat32MAXBCKFloat32MEANVARFloat32float32MEDVARFloat32float32BCKMTHDUInt8VARMTHDUInt8BACKMAGFloat32float32N_STARSUInt32MAG_95QFloat32M_CHI2DFloat32Astrometry and sky footprint (27 of 27 kept)
RAFloat64float64DECFloat64float64UPIX_PARUInt64uint64UPIX_LOWUInt64uint64UPIX_HIGUInt64uint64AST_NSRCUInt16Int32AST_ARMSFloat32float32AST_ERRMFloat32float32AST_CATLowCardinality(String)PIXSCALEFloat32ROTATFloat32RA1Float64float32RA2Float64float32RA3Float64float32RA4Float64float32DEC1Float64float32DEC2Float64float32DEC3Float64float32DEC4Float64float32RAU1Float64float32RAU2Float64float32RAU3Float64float32RAU4Float64float32DECU1Float64float32DECU2Float64float32DECU3Float64float32DECU4Float64float32WCS solution (28 of 36 kept)
CTYPE1LowCardinality(String)char8CTYPE2LowCardinality(String)WCSAXESUInt8RADESYSLowCardinality(String)LONPOLEFloat32LATPOLEFloat32CUNIT1LowCardinality(String)CUNIT2LowCardinality(String)CRPIX1Float64float32CRPIX2Float64float32CRVAL1Float64float32CRVAL2Float64float32CD1_1Float64float32CD1_2Float64float32CD2_1Float64float32CD2_2Float64float32PV1_0Float64float32PV1_1Float64float32PV1_2Float64float32PV1_4Float64float32PV1_5Float64float32PV1_6Float64float32PV1_7Float64float32PV1_8Float64float32PV1_9Float64float32PV1_10Float64float32PV2_0Float64float32PV2_1Float64float32PV2_2Float64float32PV2_4Float64float32PV2_5Float64float32PV2_6Float64float32PV2_7Float64float32PV2_8Float64float32PV2_9Float64float32PV2_10Float64float32PSF (21 of 21 kept)
FWHEFloat32FWHMFloat32float32PSF_FITNFloat32PSF_FITAFloat32PSF_FITBFloat32PSF_FITTFloat32PSF_NSTUInt16PSF_NPKUInt8PSF_PKRFloat32PSF_DPKFloat32PSF_RPKFloat32PSF_ERRFloat32PSF_S2Float32PSF_AF_1Float32PSF_AF_2Float32PSF_AF_3Float32PSF_AF_4Float32MED_X2Float32MED_Y2Float32MED_XYFloat32QUICFWHMFloat32Relative photometric calibration (PH_) (8 of 9 kept)
PH_ZPFloat32float32PH_COL1Float32float32PH_MEDCFloat32float32PH_RMSFloat32float32PH_NSRCUInt16Int32PH_MAGSYLowCardinality(String)char8PH_MAGTLowCardinality(String)char16PH_MAGTELowCardinality(String)char16PH_CATLowCardinality(String)Absolute photometric calibration (PT_): summary (23 of 23 kept)
PT_ZPFloat32PT_NORMULowCardinality(String)PT_FNORMFloat32LIMMAGFloat32float32LIMMAGERFloat32PT_RMSFloat32float32PT_ARMSFloat32PT_CHI2Float32PT_DOFUInt16PT_NCALIUInt16PT_DZPABFloat32PT_CTAFloat32PT_CTA2Float32PT_CTA3Float32PT_CTA4Float32PT_CTAEFloat32PT_AREFLowCardinality(String)PT_SPECLowCardinality(String)PT_CATLowCardinality(String)PT_REFSLFloat32PT_REFPVFloat32PT_REFCFloat32PT_CO2PPFloat32Absolute photometric calibration (PT_): transmission-model parameters (77 of 77 kept)
PT_1_NLowCardinality(String)PT_1_V1Float32PT_1_F1UInt8PT_2_NLowCardinality(String)PT_2_V1Float32PT_2_F1UInt8PT_2_V2Float32PT_2_F2UInt8PT_3_NLowCardinality(String)PT_3_V1Float32PT_3_F1UInt8PT_3_V2Float32PT_3_F2UInt8PT_3_V3Float32PT_3_F3UInt8PT_4_NLowCardinality(String)PT_4_V1Float32PT_4_F1UInt8PT_4_V2Float32PT_4_F2UInt8PT_5_NLowCardinality(String)PT_5_V1Float32PT_5_F1UInt8PT_5_V2Float32PT_5_F2UInt8PT_5_V3Float32PT_5_F3UInt8PT_6_NLowCardinality(String)PT_6_V1Float32PT_6_F1UInt8PT_6_V2Float32PT_6_F2UInt8PT_6_V3Float32PT_6_F3UInt8PT_6_V4Float32PT_6_F4UInt8PT_7_NLowCardinality(String)PT_7_V1Float32PT_7_F1UInt8PT_7_V2Float32PT_7_F2UInt8PT_7_V3Float32PT_7_F3UInt8PT_7_V4Float32PT_7_F4UInt8PT_7_V5Float32PT_7_F5UInt8PT_7_V6Float32PT_7_F6UInt8PT_7_V7Float32PT_7_F7UInt8PT_7_V8Float32PT_7_F8UInt8PT_7_V9Float32PT_7_F9UInt8PT_8_NLowCardinality(String)PT_8_V1Float32PT_8_F1UInt8PT_8_V2Float32PT_8_F2UInt8PT_8_V3Float32PT_8_F3UInt8PT_8_V4Float32PT_8_F4UInt8PT_P_NLowCardinality(String)PT_P_NX1Float32PT_P_NX2Float32PT_P_NY1Float32PT_P_NY2Float32PT_P_V1Float32PT_P_F1UInt8PT_P_V2Float32PT_P_F2UInt8PT_P_V3Float32PT_P_F3UInt8PT_P_V4Float32PT_P_F4UInt8Aperture corrections (26 of 26 kept)
APC0_A1Float32APCX_A1Float32APCY_A1Float32APCXY_A1Float32APCC_A1Float32APCCE_A1Float32APC0_A2Float32APCX_A2Float32APCY_A2Float32APCXY_A2Float32APCC_A2Float32APCCE_A2Float32APC0_A4Float32APCX_A4Float32APCY_A4Float32APCXY_A4Float32APCC_A4Float32APCCE_A4Float32APC0_PSFloat32APCX_PSFloat32APCY_PSFloat32APCXY_PSFloat32APCC_PSFloat32APCCE_PSFloat32APCC_REFFloat32APCOR_NUInt16Columns computed or corrected at ingestion
These columns cannot be copied from the header as they are. The ingestion code has to compute or convert them.
ingestion_time_jd(new, not in the header)Float64diff_srcandimage_visit. Defaultnan.id_coadd(new, not in the header; option, see note 2)UInt64ID_COADDof the coadd this epoch went into: the coadd of the same camera, visit (SUBDIR) andCROPID, whoseID_PROCF…ID_PROCLrange contains thisID_PROC. 0 if the visit has no coadd.ID_PROC,ID_RAW,ID_DARK,ID_FLATUInt64'2224430602496017002'). Convert the text directly to an integer, never through adouble. Adoubleis exact only up to about 9×10¹⁵, and the IDs are about 2×10¹⁸, so the lowest bits, which hold the image type and level, would be lost. That is also why the IDs are text in the header on purpose (imProc.db.generateImageID, since 2024): MATLAB's FITS writer rejectsuint64, and AstroPack reads numeric header values asdouble. Missing or blank → 0.UPIX_PAR,UPIX_LOW,UPIX_HIGUInt64'51198298771'); convert to an integer. Missing → 0.LIGHTSEC,OVERSCAN,CCDSEC,ORIGSEC,ORIGUSEC,UNIQSEC,EXCLSEC,ORIGESECInt16each'[xmin xmax ymin ymax]'into four numbers (imUtil.ccdsec.ccdsecStr2num), written to<keyword>_xmin,_xmax,_ymin,_ymax. Missing → 0.DATE-OBS(columndate_obs)DateTime64(3,'UTC')DATE-OBS(e.g.'2026-04-06T00:39:34.229') to a date-time with milliseconds.VERSIONUInt8'1'); convert to an integer.FIELDIDLowCardinality(String)1214.), named targets as text ('21109WD'). Store the number without the trailing point ('1214'), so that it matchesOBJECT.NODENUMB,MOUNTNUM,CAMNUM,CROPID,COUNTER,CAMMODE,CAMGAIN,CAMOFFS,BINX,BINY,N_STARS,AST_NSRC,PSF_NST,PSF_NPK,PH_NSRC,PT_DOF,PT_NCALI,PT_1_F1,PT_2_F1,PT_2_F2,PT_3_F1,PT_3_F2,PT_3_F3,PT_4_F1,PT_4_F2,PT_5_F1,PT_5_F2,PT_5_F3,PT_6_F1,PT_6_F2,PT_6_F3,PT_6_F4,PT_7_F1,PT_7_F2,PT_7_F3,PT_7_F4,PT_7_F5,PT_7_F6,PT_7_F7,PT_7_F8,PT_7_F9,PT_8_F1,PT_8_F2,PT_8_F3,PT_8_F4,PT_P_F1,PT_P_F2,PT_P_F3,PT_P_F4,APCOR_NUInt8/UInt16/UInt32double, so the header writer gives them a trailing point (1.), and writes a blank card when the value is NaN (e.g. failed photometric calibration). Convert to an integer; blank → 0. (BCKMTHD,VARMTHDandPSTATUSare set as integer types and written as integers already.)Float32/Float64columnsnan. Seen in the sample:PSF_PKR(11448),PSF_DPK(11448),PT_ARMS(480),PT_RMS(100),PT_CHI2(20),PT_DZPAB(20),PT_CTA(20),PT_CTA2(20) and 10 more keywords. ThePT_blanks are the coadd's: they appear in all epochs of a coadd sub-image whose calibration is blank.Notes on specific choices
PT_) values except the normalisationPT_1_V1, and all aperture corrections (APC*), are the coadd's values: in the sample,PT_ZP,PT_RMSand the transmission parameters of an epoch equal those of its coadd (except in one sub-image, 20 epochs, where the coadd's are blank), whilePT_1_V1,LIMMAGand the relative calibrationPH_*differ from epoch to epoch. These 121 columns are kept for now; whether to drop them and take them fromimage_visitthroughid_coadd(note 2) will be discussed later. The camera, mount and software keywords that repeat the coadd are needed inimage_procand stay.id_coaddlink. The epoch header has noID_COADD, soimage_procgets anid_coaddcolumn computed at ingestion: the coadd header of the same visit and sub-image hasID_PROCFandID_PROCL, theID_PROCof its first and last epoch (both matched an epochID_PROCin all 624 coadd headers of the sample). A range onID_PROCalone is not enough, because the IDs of the other sub-images of the same visit fall in the same range; the crop number (CROPID) must match too.LEVELis wrong in the epoch header. It saysrawin all 12,216 headers, although the files areprocproducts. TheID_PROCvalues carry the same wrong level (raw) in their level bits. The IDs are still unique (ID_RAWdiffers in the crop number and the time), so this does not block the table, butLEVELis ❌ here (it is fixed for this table) and the level bits ofid_procshould not be relied on until this is fixed in the pipeline (v1 epoch images keep LEVEL = raw, so ID_PROC has the raw level #1412).OPT_CLN(date of the last optics cleaning) is not in the epoch headers at all.image_proc:FILENAME,PROJNAME,SUBDIR,DIRYEAR,DIRMON,DIRDAYandFILETIME. They are kept inimage_visit, and the link to it isid_coadd.Differences from the 2024 design sheet
proc_imagescolumn of the 2024 sheet: mostly PSF quality, absolute photometry (PT_*,APC*), image IDs, source counts and the processing status mask, as in visit_images: a ClickHouse table for LAST v1 coadd headers #1409.NAXIS1/2,CCDID,CTYPE1andPH_MAGTE(the same as in visit_images: a ClickHouse table for LAST v1 coadd headers #1409), and the file location keywordsSUBDIR,DIRYEAR,DIRMON,DIRDAYandFILETIME(note 5).Float64instead offloat32forCRVAL,CD,PVand the corner coordinates.Indexes
Draft for @chentishler to approve or edit.
discussion2sqlreads this table intoCREATE TABLEand into an ALTER script for a table that already exists (types:minmax,set(N),bloom_filter,tokenbf_v1,ngrambf_v1; granularity in blocks of 8192 rows; an empty table cell means this table).idx_jdjdminmaxidx_jd_startjd_startminmaxidx_midjdmidjdminmaxidx_date_obsdate_obsminmaxidx_ingestingestion_time_jdminmaxidx_id_coaddid_coaddminmaximage_visitrow)idx_id_rawid_rawminmaxidx_id_darkid_darkbloom_filter(0.01)idx_id_flatid_flatbloom_filter(0.01)idx_fieldidfieldidbloom_filter(0.01)idx_objectobjectbloom_filter(0.01)idx_upix_parupix_parbloom_filter(0.01)idx_pipeverpipeverset(16)Index choices by column
The table is sorted by
id_proc, whose highest bits are the epoch time (MIDJD, checked on all 98,352 rows inlast_test;ID_COADDandID_RAWhave the same layout with the coadd and raw time,ID_DARKandID_FLATthe time of the calibration image). So each block of 8192 rows holds about 4 minutes of all cameras, i.e. about one epoch of 12 visits. A skip index helps only for a column whose values are bunched within such a block: time columns and the time-ordered IDs (minmax), and values that occur in few blocks, such as one field or one flat (bloom_filter). At about 1.6×10⁸ rows (20,000 blocks) per year the table is 20 timesimage_visit, so a column without an index takes up to about a second to read.Measured on a simulation of this table (no multi-camera epoch table exists yet): every coadd row of
last.visit_imagesof the last 60 days made into 20 epochs 20 s apart, sorted likeid_proc, 6190 blocks; per block: 33 cameras, 12 fields, 28upix_parvalues, 78° in Dec:id_coadd)fieldid)id_flat)upix_par)Columns without an index (every ✅ column not in the table above):
id_procnodenumb,mountnum,camnum,cropid,camnametime,lst,crdatelstrepeats every day;crdateandtimefollow the reduction, not the observation.m_ra,m_ha,m_dec,m_az,m_alt,m_jra,m_jdec,m_ara,m_aha,m_adec,m_adra,m_adha,m_addec,ra,dec,upix_low,upix_hig,ra1,ra2,ra3,ra4,dec1,dec2,dec3,dec4,rau1,rau2,rau3,rau4,decu1,decu2,decu3,decu4upix_par(question 1).version,gitastrp,git_camr,gitfocus,bias_im,flat_im,pstatus,schedulr,exptime,expmode,counter,filter,obslon,obslat,obsalt,trk_ra,trk_dec,camtemp,camcool,cammode,camgain,camoffs,origgain,darkcur,saturval,nonlin,binx,biny,lightsec_xmin,lightsec_xmax,lightsec_ymin,lightsec_ymax,overscan_xmin,overscan_xmax,overscan_ymin,overscan_ymax,ccdsec_xmin,ccdsec_xmax,ccdsec_ymin,ccdsec_ymax,origsec_xmin,origsec_xmax,origsec_ymin,origsec_ymax,origusec_xmin,origusec_xmax,origusec_ymin,origusec_ymax,uniqsec_xmin,uniqsec_xmax,uniqsec_ymin,uniqsec_ymax,exclsec_xmin,exclsec_xmax,exclsec_ymin,exclsec_ymax,origesec_xmin,origesec_xmax,origesec_ymin,origesec_ymax,bckmthd,varmthd,ast_cat,psf_npk,psf_rpk,ph_magsy,ph_magt,ph_cat,pt_normu,pt_aref,pt_spec,pt_cat,pt_refsl,pt_refpv,pt_refc,pt_co2pp,pt_1_n,pt_1_f1,pt_2_n,pt_2_f1,pt_2_f2,pt_3_n,pt_3_f1,pt_3_f2,pt_3_f3,pt_4_n,pt_4_f1,pt_4_f2,pt_5_n,pt_5_f1,pt_5_f2,pt_5_f3,pt_6_n,pt_6_f1,pt_6_f2,pt_6_f3,pt_6_f4,pt_7_n,pt_7_f1,pt_7_f2,pt_7_f3,pt_7_f4,pt_7_f5,pt_7_f6,pt_7_f7,pt_7_f8,pt_7_f9,pt_8_n,pt_8_f1,pt_8_f2,pt_8_f3,pt_8_f4,pt_p_n,pt_p_nx1,pt_p_nx2,pt_p_ny1,pt_p_ny2,pt_p_f1,pt_p_f2,pt_p_f3,pt_p_f4,apcc_refairmass,mnttemp,focus,prvfocus,gain,readnoi,meanbck,medbck,stdbck,minbck,maxbck,meanvar,medvar,backmag,n_stars,mag_95q,m_chi2d,ast_nsrc,ast_arms,ast_errm,pixscale,rotat,crpix1,crpix2,crval1,crval2,cd1_1,cd1_2,cd2_1,cd2_2,pv1_0,pv1_1,pv1_2,pv1_4,pv1_5,pv1_6,pv1_7,pv1_8,pv1_9,pv1_10,pv2_0,pv2_1,pv2_2,pv2_4,pv2_5,pv2_6,pv2_7,pv2_8,pv2_9,pv2_10,fwhe,fwhm,psf_fitn,psf_fita,psf_fitb,psf_fitt,psf_nst,psf_pkr,psf_dpk,psf_err,psf_s2,psf_af_1,psf_af_2,psf_af_3,psf_af_4,med_x2,med_y2,med_xy,quicfwhm,ph_zp,ph_col1,ph_medc,ph_rms,ph_nsrc,pt_zp,pt_fnorm,limmag,limmager,pt_rms,pt_arms,pt_chi2,pt_dof,pt_ncali,pt_dzpab,pt_cta,pt_cta2,pt_cta3,pt_cta4,pt_ctae,pt_1_v1,pt_2_v1,pt_2_v2,pt_3_v1,pt_3_v2,pt_3_v3,pt_4_v1,pt_4_v2,pt_5_v1,pt_5_v2,pt_5_v3,pt_6_v1,pt_6_v2,pt_6_v3,pt_6_v4,pt_7_v1,pt_7_v2,pt_7_v3,pt_7_v4,pt_7_v5,pt_7_v6,pt_7_v7,pt_7_v8,pt_7_v9,pt_8_v1,pt_8_v2,pt_8_v3,pt_8_v4,pt_p_v1,pt_p_v2,pt_p_v3,pt_p_v4,apc0_a1,apcx_a1,apcy_a1,apcxy_a1,apcc_a1,apcce_a1,apc0_a2,apcx_a2,apcy_a2,apcxy_a2,apcc_a2,apcce_a2,apc0_a4,apcx_a4,apcy_a4,apcxy_a4,apcc_a4,apcce_a4,apc0_ps,apcx_ps,apcy_ps,apcxy_ps,apcc_ps,apcce_ps,apcor_nQuestions for @chentishler:
bloom_filteronupix_par(9% of the blocks for one partition) or a projection sorted by position,SELECT * ORDER BY (upix_par, id_proc), which reads about one block per partition but doubles the size of the table (tens of GB per year here).(nodenumb, mountnum, camnum, id_proc)would read only that camera, at twice the size.bloom_filteronLowCardinality(String)columns and projections; both work on 26.6 (euclid).minmaxonfwhm(and similar values) skips some blocks (82% for FWHM < 2.5 in the simulation). Worth adding for the most used quality cuts, once real epoch data can be measured?All reactions