Repository navigation
Replies: 7 comments 1 reply
|
table name: I suggest image_visit |
|
comments: M_A*, M_AD*, are needed for polar alignment... so add to table. I would add APCC_REF Why are there many granularity 1? where do I find the suggested indexing scheme? |
|
@EranOfek, a first test of this table is ready. The table was created from the column choices in this post, and the headers of 3 visits (72 coadd sub-images) were loaded into The two new functions are pushed to |
|
@chentishler, I added a draft of the |
|
@chentishler The script that creates |
|
@chentishler Before we start filling Proposal, with everything else as you created it (columns, indexes, projections, ENGINE = ReplacingMergeTree(ingestion_time_jd)
SETTINGS deduplicate_merge_projection_mode = 'rebuild', storage_policy = 'bigdata3', index_granularity = 8192
Only the v1 reduction of each visit will be ingested ( Could you make the change, or would you rather we re-create the two tables ourselves from your |
|
I will check how to do both replacing and replicating, I will fix it later
today.
Chen
…On Tue, Nov 10, 2026, 21:54 Alexander Krassilchtchikov < ***@***.***> wrote:
@chentishler <https://github.com/chentishler> Before we start filling
last1.image_visit and last1.image_proc, we would like both tables to use
ReplacingMergeTree instead of MergeTree, so that a visit inserted twice
(for example after a lost .status stamp) leaves one row per image, not
two.
Proposal, with everything else as you created it (columns, indexes,
projections, ORDER BY, storage_policy):
ENGINE = ReplacingMergeTree(ingestion_time_jd)
SETTINGS deduplicate_merge_projection_mode = 'rebuild', storage_policy = 'bigdata3', index_granularity = 8192
- Duplicates are rows with the same ORDER BY key (id_coadd / id_proc);
ingestion_time_jd makes the latest insert the one that is kept.
- ClickHouse 26.6 refuses projections on a ReplacingMergeTree table
unless deduplicate_merge_projection_mode is set; 'rebuild' keeps your
projections consistent with the remaining rows.
- The tables are still empty, so it is a drop and re-create.
Only the v1 reduction of each visit will be ingested (v2/v3 dirs are
skipped, d995d04
<d995d04>
).
Could you make the change, or would you rather we re-create the two tables
ourselves from your SHOW CREATE TABLE with only these lines changed?
Either way, please update your master files in astro-clickhouse.
—
Reply to this email directly, view it on GitHub
<#1409?email_source=notifications&email_token=AUPHYZWEILLCUAYJJI2LYR35YNW55A5CNFSNUABIM5UWIORPF5TWS5BNNB2WEL2ENFZWG5LTONUW63SDN5WW2ZLOOQXTCOJVGI3TONBRUZZGKYLTN5XKO3LFNZ2GS33OUVSXMZLOOSWGM33PORSXEX3DNRUWG2Y#discussioncomment-19527741>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AUPHYZV5POOQECFWCESYI635YNW55AVCNFSNUABIKJSXA33TNF2G64TZHMZTINZZGE4DCOJSHNCGS43DOVZXG2LPNY5TCMJQGQZDQOBWUF3AE>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
We want a new ClickHouse table,
image_visit, that keeps the header of every LAST v1 coadd image (one row per coadd sub-image, i.e. 24 rows per visit), so image quality, calibration and pointing can be queried without opening FITS files. Below is a first proposal: for every header keyword (except HISTORY and COMMENT), whether to keep it as a column, its type, default value, the storage resolution it needs, and how many distinct values to expect. Please comment on the ✅/❌ choices.Related: #1411 proposes the source tables
src_visitandsrc_proc, and #1413 the epoch image tableimage_proc, in the same format. Once this proposal is settled, it will be used to create the actual ClickHouse tableimage_visit.What the proposal is based on
LAST.01.02.01_20260406.004254.291_clear_1214_000_001_005_sci_coadd_Image_1.fits(last02e, pipelinelv1.0.4).21109WD) have the same keyword set.MAXBCKD, the largest difference of the background level (MEDBCK) between the images of the visit (findTransients: PSF-orientation step crashes when the binarized PSF has several (or no) regions #1355). It was added to the coadd header after the sample was taken, so older coadds do not have it and get the defaultnan.lv1.1.2(checked on the products of the night 2026-10-20 on euclid).SCHEDULR(the scheduler that gave the target, e.g.last.survey, LAST_OCS Background estimation can be improved #132) andQUICFWHM(the camera software's quick FWHM estimate in the centre of the raw image, in pixels) come from the raw image header.PT_FNORMandPT_NORMUcome from the absolute calibration (PhotCalibTrans: drop the geometric collecting area #1431):PT_NORMU = 'm2'says that the normalisationPT_1_V1is now an effective collecting area in m² (about 0.019), andPT_FNORMis that area divided by the telescope area π·0.1397² m² (about 0.31). In older headers (noPT_NORMU)PT_1_V1is already relative to the telescope area (about 0.35), so it is comparable toPT_FNORM, not to the newPT_1_V1. Older products do not have the four keywords and get the defaults.Float32, about 7 significant digits) and 8-byte (Float64) numbers.Design-Database-Pipeline-ClickHouse.xlsx, tab Images,visit_imagescolumn, updated 18/11/2024) already has this column, and with which name and type.SIMPLE,NAXIS*,CTYPE*,CUNIT*, …), fixed by definition for a coadd (LEVEL,IMTYPE,SUBLEVEL), constant for the site (TIMEZONE,EQUINOX,SPATNBIN), or an exact copy or simple function of another kept keyword:FULLPROJrepeatsPROJNAME,ID_PROCrepeatsID_PROCF,CCDIDis always 1, andPH_MAGTEfollows fromPH_MAGT. Keywords that are constant in the sample but can change between pipeline versions or hardware (catalogue names, methods,NODENUMB) are kept: a constant column costs almost nothing to store.-replaced by_(DATE-OBSbecomesdate_obs). This matches the style of the existinglast.diff_srctable. IDs and theUPIX_*values becomeUInt64. Indiff_srcthe IDs areInt64, so a join between the two tables on an ID needs a cast. Each image-section keyword (*SECandOVERSCAN, header text'[xmin xmax ymin ymax]') becomes fourInt16columns,<keyword>_xmin,_xmax,_yminand_ymax, as in the existinglast.visit_imagestable on euclid.Sorting and indexes
The table is sorted by
id_coadd(ORDER BY id_coadd). The observation time sits in the highest bits of the ID, so this order is close to time order. Additional indexes will be defined by @chentishler.Keyword table
Click a group to open it.
FITS structure (0 of 9 kept)
SIMPLEUInt8BITPIXInt8NAXISUInt8NAXIS1UInt16int16NAXIS2UInt16int16EXTENDUInt8BZEROFloat32BSCALEFloat32LONGSTRNLowCardinality(String)Identification, files and versions (26 of 31 kept)
FILENAMEStringPROJNAMELowCardinality(String)FULLPROJLowCardinality(String)NODENUMBUInt8int16MOUNTNUMUInt8int8CAMNUMUInt8int8CROPIDUInt8int16CCDIDUInt8int16LEVELLowCardinality(String)IMTYPELowCardinality(String)VERSIONUInt8SUBDIRStringchar16DIRYEARUInt16int16DIRMONUInt8int8DIRDAYUInt8int8FILETIMEStringstringTIMEStringPIPEVERLowCardinality(String)char32GITASTRPLowCardinality(String)GIT_CAMRLowCardinality(String)GITFOCUSLowCardinality(String)ID_COADDUInt64ID_PROCUInt64ID_PROCFUInt64ID_PROCLUInt64ID_RAWUInt64BIAS_IMLowCardinality(String)ID_DARKUInt64int64FLAT_IMLowCardinality(String)ID_FLATUInt64int64PSTATUSUInt32Target and observation time (16 of 16 kept)
OBJECTLowCardinality(String)char16FIELDIDLowCardinality(String)char16SCHEDULRLowCardinality(String)DATE-OBSDateTime64(3,'UTC')DateTime64(3)JDFloat64float64JD_STARTFloat64MIDJDFloat64float64MINJDFloat64MAXJDFloat64LSTFloat32float64CRDATEDateTime64(3,'UTC')EXPTIMEFloat32float32EXPMODELowCardinality(String)COUNTERUInt32int16FILTERLowCardinality(String)char8AIRMASSFloat32float32Observatory and mount (22 of 24 kept)
TIMEZONEFloat32OBSLONFloat64float32OBSLATFloat64float32OBSALTFloat32float32EQUINOXFloat32M_RAFloat32float32M_HAFloat32float32M_DECFloat32float32M_AZFloat32M_ALTFloat32M_JRAFloat32float32M_JDECFloat32float32M_ARAFloat32float32M_AHAFloat32float32M_ADECFloat32float32M_ADRAFloat32float32M_ADHAFloat32float32M_ADDECFloat32float32TRK_RAFloat32TRK_DECFloat32MNTTEMPFloat32float32FOCUSFloat32Int32PRVFOCUSFloat32Int32OPT_CLNFloat64float32Camera and detector (22 of 22 kept)
CAMNAMELowCardinality(String)CAMTEMPFloat32CAMCOOLFloat32float32CAMMODEUInt8CAMGAINUInt16float32CAMOFFSUInt16float32GAINFloat32ORIGGAINFloat32float32READNOIFloat32float32DARKCURFloat32float32SATURVALFloat32float32NONLINFloat32float32BINXUInt8float32BINYUInt8float32LIGHTSECInt16:lightsec_xmin,_xmax,_ymin,_ymaxOVERSCANInt16:overscan_xmin,_xmax,_ymin,_ymaxCCDSECInt16:ccdsec_xmin,_xmax,_ymin,_ymaxORIGSECInt16:origsec_xmin,_xmax,_ymin,_ymaxint16ORIGUSECInt16:origusec_xmin,_xmax,_ymin,_ymaxint16UNIQSECInt16:uniqsec_xmin,_xmax,_ymin,_ymaxint16EXCLSECInt16:exclsec_xmin,_xmax,_ymin,_ymaxORIGESECInt16:origesec_xmin,_xmax,_ymin,_ymaxCoaddition (5 of 6 kept)
NCOADDUInt8int16COADDOPLowCardinality(String)char16AVNCOADDFloat32float32MINCOADDUInt8MAXBCKDFloat32SUBLEVELLowCardinality(String)Background and image statistics (17 of 18 kept)
MEANBCKFloat32float32MEDBCKFloat32float32STDBCKFloat32float32MINBCKFloat32MAXBCKFloat32MEANVARFloat32float32MEDVARFloat32float32BCKMTHDUInt8VARMTHDUInt8BACKMAGFloat32float32N_STARSUInt32MAG_95QFloat32M_CHI2DFloat32SPATQ50Float32SPATQ75Float32SPATQ90Float32SPATQ95Float32SPATNBINUInt16Astrometry and sky footprint (33 of 33 kept)
RAFloat64float64DECFloat64float64UPIX_PARUInt64uint64UPIX_LOWUInt64uint64UPIX_HIGUInt64uint64AST_NSRCUInt16Int32AST_ARMSFloat32float32AST_ERRMFloat32float32AST_CATLowCardinality(String)RP_MRMSFloat32RP_MMRMSFloat32PIXSCALEFloat32ROTATFloat32RA1Float64float32RA2Float64float32RA3Float64float32RA4Float64float32DEC1Float64float32DEC2Float64float32DEC3Float64float32DEC4Float64float32RAU1Float64float32RAU2Float64float32RAU3Float64float32RAU4Float64float32DECU1Float64float32DECU2Float64float32DECU3Float64float32DECU4Float64float32GM_RATEXFloat32float32GM_STDXFloat32float32GM_RATEYFloat32float32GM_STDYFloat32float32WCS solution (28 of 36 kept)
CTYPE1LowCardinality(String)char8CTYPE2LowCardinality(String)WCSAXESUInt8RADESYSLowCardinality(String)LONPOLEFloat32LATPOLEFloat32CUNIT1LowCardinality(String)CUNIT2LowCardinality(String)CRPIX1Float64float32CRPIX2Float64float32CRVAL1Float64float32CRVAL2Float64float32CD1_1Float64float32CD1_2Float64float32CD2_1Float64float32CD2_2Float64float32PV1_0Float64float32PV1_1Float64float32PV1_2Float64float32PV1_4Float64float32PV1_5Float64float32PV1_6Float64float32PV1_7Float64float32PV1_8Float64float32PV1_9Float64float32PV1_10Float64float32PV2_0Float64float32PV2_1Float64float32PV2_2Float64float32PV2_4Float64float32PV2_5Float64float32PV2_6Float64float32PV2_7Float64float32PV2_8Float64float32PV2_9Float64float32PV2_10Float64float32PSF (22 of 22 kept)
FWHEFloat32FWHMFloat32float32PSF_FITNFloat32PSF_FITAFloat32PSF_FITBFloat32PSF_FITTFloat32PSF_NSTUInt16PSF_NPKUInt8PSF_PKRFloat32PSF_DPKFloat32PSF_RPKFloat32PSF_ERRFloat32PSF_S2Float32PSF_AF_1Float32PSF_AF_2Float32PSF_AF_3Float32PSF_AF_4Float32MED_X2Float32MED_Y2Float32MED_XYFloat32SNCHI2Float32QUICFWHMFloat32Relative photometric calibration (PH_) (8 of 9 kept)
PH_ZPFloat32float32PH_COL1Float32float32PH_MEDCFloat32float32PH_RMSFloat32float32PH_NSRCUInt16Int32PH_MAGSYLowCardinality(String)char8PH_MAGTLowCardinality(String)char16PH_MAGTELowCardinality(String)char16PH_CATLowCardinality(String)Absolute photometric calibration (PT_): summary (23 of 23 kept)
PT_ZPFloat32PT_NORMULowCardinality(String)PT_FNORMFloat32LIMMAGFloat32float32LIMMAGERFloat32PT_RMSFloat32float32PT_ARMSFloat32PT_CHI2Float32PT_DOFUInt16PT_NCALIUInt16PT_DZPABFloat32PT_CTAFloat32PT_CTA2Float32PT_CTA3Float32PT_CTA4Float32PT_CTAEFloat32PT_AREFLowCardinality(String)PT_SPECLowCardinality(String)PT_CATLowCardinality(String)PT_REFSLFloat32PT_REFPVFloat32PT_REFCFloat32PT_CO2PPFloat32Absolute photometric calibration (PT_): transmission-model parameters (77 of 77 kept)
PT_1_NLowCardinality(String)PT_1_V1Float32PT_1_F1UInt8PT_2_NLowCardinality(String)PT_2_V1Float32PT_2_F1UInt8PT_2_V2Float32PT_2_F2UInt8PT_3_NLowCardinality(String)PT_3_V1Float32PT_3_F1UInt8PT_3_V2Float32PT_3_F2UInt8PT_3_V3Float32PT_3_F3UInt8PT_4_NLowCardinality(String)PT_4_V1Float32PT_4_F1UInt8PT_4_V2Float32PT_4_F2UInt8PT_5_NLowCardinality(String)PT_5_V1Float32PT_5_F1UInt8PT_5_V2Float32PT_5_F2UInt8PT_5_V3Float32PT_5_F3UInt8PT_6_NLowCardinality(String)PT_6_V1Float32PT_6_F1UInt8PT_6_V2Float32PT_6_F2UInt8PT_6_V3Float32PT_6_F3UInt8PT_6_V4Float32PT_6_F4UInt8PT_7_NLowCardinality(String)PT_7_V1Float32PT_7_F1UInt8PT_7_V2Float32PT_7_F2UInt8PT_7_V3Float32PT_7_F3UInt8PT_7_V4Float32PT_7_F4UInt8PT_7_V5Float32PT_7_F5UInt8PT_7_V6Float32PT_7_F6UInt8PT_7_V7Float32PT_7_F7UInt8PT_7_V8Float32PT_7_F8UInt8PT_7_V9Float32PT_7_F9UInt8PT_8_NLowCardinality(String)PT_8_V1Float32PT_8_F1UInt8PT_8_V2Float32PT_8_F2UInt8PT_8_V3Float32PT_8_F3UInt8PT_8_V4Float32PT_8_F4UInt8PT_P_NLowCardinality(String)PT_P_NX1Float32PT_P_NX2Float32PT_P_NY1Float32PT_P_NY2Float32PT_P_V1Float32PT_P_F1UInt8PT_P_V2Float32PT_P_F2UInt8PT_P_V3Float32PT_P_F3UInt8PT_P_V4Float32PT_P_F4UInt8Aperture corrections (26 of 26 kept)
APC0_A1Float32APCX_A1Float32APCY_A1Float32APCXY_A1Float32APCC_A1Float32APCCE_A1Float32APC0_A2Float32APCX_A2Float32APCY_A2Float32APCXY_A2Float32APCC_A2Float32APCCE_A2Float32APC0_A4Float32APCX_A4Float32APCY_A4Float32APCXY_A4Float32APCC_A4Float32APCCE_A4Float32APC0_PSFloat32APCX_PSFloat32APCY_PSFloat32APCXY_PSFloat32APCC_PSFloat32APCCE_PSFloat32APCC_REFFloat32APCOR_NUInt16Columns computed or corrected at ingestion
These columns cannot be copied from the header as they are. The ingestion code has to compute or convert them.
ingestion_time_jd(new, not in the header)Float64diff_src. Defaultnan.ID_COADD,ID_PROCF,ID_PROCL,ID_RAW,ID_DARK,ID_FLATUInt64'2224430602496017002'). Convert the text directly to an integer, never through adouble. Adoubleis exact only up to about 9×10¹⁵, and the IDs are about 2×10¹⁸, so the lowest bits, which hold the image type and level, would be lost. That is also why the IDs are text in the header on purpose (imProc.db.generateImageID, since 2024): MATLAB's FITS writer rejectsuint64, and AstroPack reads numeric header values asdouble. Missing or blank → 0.UPIX_PAR,UPIX_LOW,UPIX_HIGUInt64'51198298771'); convert to an integer. Missing → 0.LIGHTSEC,OVERSCAN,CCDSEC,ORIGSEC,ORIGUSEC,UNIQSEC,EXCLSEC,ORIGESECInt16each'[xmin xmax ymin ymax]'into four numbers (imUtil.ccdsec.ccdsecStr2num), written to<keyword>_xmin,_xmax,_ymin,_ymax. Missing → 0.DATE-OBS(columndate_obs)DateTime64(3,'UTC')DATE-OBS(e.g.'2026-04-06T00:39:34.229') to a date-time with milliseconds.VERSIONUInt8'1'); convert to an integer.FIELDIDLowCardinality(String)1214.), named targets as text ('21109WD'). Store the number without the trailing point ('1214'), so that it matchesOBJECT.NODENUMB,MOUNTNUM,CAMNUM,CROPID,DIRYEAR,DIRMON,DIRDAY,COUNTER,CAMMODE,CAMGAIN,CAMOFFS,BINX,BINY,NCOADD,MINCOADD,N_STARS,AST_NSRC,PSF_NST,PSF_NPK,PH_NSRC,PT_DOF,PT_NCALI,PT_1_F1,PT_2_F1,PT_2_F2,PT_3_F1,PT_3_F2,PT_3_F3,PT_4_F1,PT_4_F2,PT_5_F1,PT_5_F2,PT_5_F3,PT_6_F1,PT_6_F2,PT_6_F3,PT_6_F4,PT_7_F1,PT_7_F2,PT_7_F3,PT_7_F4,PT_7_F5,PT_7_F6,PT_7_F7,PT_7_F8,PT_7_F9,PT_8_F1,PT_8_F2,PT_8_F3,PT_8_F4,PT_P_F1,PT_P_F2,PT_P_F3,PT_P_F4,APCOR_NUInt8/UInt16/UInt32double, so the header writer gives them a trailing point (1.), and writes a blank card when the value is NaN (e.g. failed photometric calibration). Convert to an integer; blank → 0. (BCKMTHD,VARMTHDandPSTATUSare set as integer types and written as integers already.)Float32/Float64columnsnan. Seen in the sample:PSF_PKR(1211),PSF_DPK(1211),RP_MRMS(14),RP_MMRMS(14),PT_ARMS(10),PT_RMS(1),PT_CHI2(1),PT_ZP(1) and 69 more keywords, all blank in the same failed-calibration sub-image.OPT_CLN, see note 4) → the column default.Notes on specific choices
PT_transmission model. AllPT_keywords are kept, including the model names (PT_k_N), the fixed inputs and the fit flags (PT_k_Fn), so the full calibration model of each image can be rebuilt from the table. Four values are fitted in production:PT_1_V1(normalisation),PT_3_V2(aerosol),PT_5_V2(water vapour) andPT_8_V2(detector QE centre), plus the spatial termsPT_P_V2..V4. Some inputs repeat other columns:PT_2_V1is the zenith angle andPT_6_V2isMNTTEMP.CRPIX,CRVAL,CD,PV, 30 columns) is marked ✅. With it, the WCS of any sub-image can be rebuilt from the table alone. If we only need footprints, the corner coordinates (RA1..4/DEC1..4,RAU*/DECU*) are enough, and these 30 columns can be dropped.Float32. At RA ≈ 250° a 4-byte number resolves about 0.1″, which is far below the pointing accuracy. The apparent mount coordinates (M_ARA,M_AHA,M_ADEC,M_ADRA,M_ADHA,M_ADDEC) are kept because they are needed for polar alignment. The sub-image centreRA/DECand the corners areFloat64, because they may be used for position searches.OPT_CLN(date of the last optics cleaning, from the camera configuration) is in every raw image of LAST.01.03.01 that night, but in only 1 of that camera's 600 coadd headers. It is kept, with NaN as the default; why the coadd header loses it will be checked separately.PSF_PKR/PSF_DPKare blank unless the PSF has more than one peak (13 of 1224 sub-images in the sample). They are kept because, together withPSF_NPK/PSF_RPK, they flag distorted (e.g. donut) PSFs.Differences from the 2024 design sheet
PT_*,APC*), image IDs, source counts and the processing status mask.NAXIS1/2,CCDID,CTYPE1andPH_MAGTE. The 2024 sheet has a single columnctype, filled fromCTYPE1with the projection name only (TPV, the same for both axes); hereCTYPE1andCTYPE2are both ❌ because their values never change (RA---TPV,DEC--TPV).CRVAL,CD,PVand the corner coordinates asfloat32. At RA ≈ 250° afloat32resolves only about 0.05″. That is comparable to the astrometric RMS (AST_ARMS0.01–0.05″ in the sample), soFloat64is proposed for these columns.Indexes
Draft for @chentishler to approve or edit.
discussion2sqlreads this table intoCREATE TABLEand into an ALTER script for a table that already exists (types:minmax,set(N),bloom_filter,tokenbf_v1,ngrambf_v1; granularity in blocks of 8192 rows; an empty table cell means this table).idx_jdjdminmaxidx_jd_startjd_startminmaxidx_midjdmidjdminmaxidx_minjdminjdminmaxidx_maxjdmaxjdminmaxidx_date_obsdate_obsminmaxidx_ingestingestion_time_jdminmaxidx_id_procfid_procfminmaxid_procf <= X AND id_procl >= X)idx_id_proclid_proclminmaxidx_id_rawid_rawminmaxidx_id_darkid_darkbloom_filter(0.01)idx_id_flatid_flatbloom_filter(0.01)idx_fieldidfieldidbloom_filter(0.01)idx_objectobjectbloom_filter(0.01)idx_upix_parupix_parbloom_filter(0.01)idx_pipeverpipeverset(16)Index choices by column
The table is sorted by
id_coadd, whose highest bits are the coadd time (MIDJD, checked on all 4920 rows inlast_test;ID_PROCF,ID_PROCLandID_RAWhave the same layout,ID_DARKandID_FLATcarry the time of the calibration image). So each block of 8192 rows holds about 1.3 hours of all cameras. A skip index helps only for a column whose values are bunched within such a block: time columns and the time-ordered IDs (minmax), and values that occur in few blocks, such as one field or one flat (bloom_filter). The table is small (about 8 million rows, 1000 blocks per year), so a query on a column without an index reads the whole column in a fraction of a second.Measured on
last.visit_images(v0 coadds of all cameras, last year, 1455 blocks, sorted by time; per block: 37 cameras, 69 fields, 77upix_parvalues, 96° in Dec):fieldid)id_flat)upix_par)Columns without an index (every ✅ column not in the table above):
id_coaddprojname,nodenumb,mountnum,camnum,cropid,camnamesubdir,diryear,dirmon,dirday,filetime,time,lst,crdatelstrepeats every day;crdateandfiletime/timefollow the reduction, not the observation;diryear/dirmon/dirday/subdirfollowjd, usejd.filenameprojname,filetime,filter,fieldid,counterandcropid. To find the row of one file, select onprojnameandfiletime.m_ra,m_ha,m_dec,m_az,m_alt,m_jra,m_jdec,m_ara,m_aha,m_adec,m_adra,m_adha,m_addec,ra,dec,upix_low,upix_hig,ra1,ra2,ra3,ra4,dec1,dec2,dec3,dec4,rau1,rau2,rau3,rau4,decu1,decu2,decu3,decu4upix_par(question 1).version,gitastrp,git_camr,gitfocus,bias_im,flat_im,pstatus,schedulr,exptime,expmode,counter,filter,obslon,obslat,obsalt,trk_ra,trk_dec,opt_cln,camtemp,camcool,cammode,camgain,camoffs,origgain,darkcur,saturval,nonlin,binx,biny,lightsec_xmin,lightsec_xmax,lightsec_ymin,lightsec_ymax,overscan_xmin,overscan_xmax,overscan_ymin,overscan_ymax,ccdsec_xmin,ccdsec_xmax,ccdsec_ymin,ccdsec_ymax,origsec_xmin,origsec_xmax,origsec_ymin,origsec_ymax,origusec_xmin,origusec_xmax,origusec_ymin,origusec_ymax,uniqsec_xmin,uniqsec_xmax,uniqsec_ymin,uniqsec_ymax,exclsec_xmin,exclsec_xmax,exclsec_ymin,exclsec_ymax,origesec_xmin,origesec_xmax,origesec_ymin,origesec_ymax,ncoadd,coaddop,mincoadd,bckmthd,varmthd,ast_cat,psf_npk,psf_rpk,ph_magsy,ph_magt,ph_cat,pt_normu,pt_aref,pt_spec,pt_cat,pt_refsl,pt_refpv,pt_refc,pt_co2pp,pt_1_n,pt_1_f1,pt_2_n,pt_2_f1,pt_2_f2,pt_3_n,pt_3_f1,pt_3_f2,pt_3_f3,pt_4_n,pt_4_f1,pt_4_f2,pt_5_n,pt_5_f1,pt_5_f2,pt_5_f3,pt_6_n,pt_6_f1,pt_6_f2,pt_6_f3,pt_6_f4,pt_7_n,pt_7_f1,pt_7_f2,pt_7_f3,pt_7_f4,pt_7_f5,pt_7_f6,pt_7_f7,pt_7_f8,pt_7_f9,pt_8_n,pt_8_f1,pt_8_f2,pt_8_f3,pt_8_f4,pt_p_n,pt_p_nx1,pt_p_nx2,pt_p_ny1,pt_p_ny2,pt_p_f1,pt_p_f2,pt_p_f3,pt_p_f4,apcc_refairmass,mnttemp,focus,prvfocus,gain,readnoi,avncoadd,maxbckd,meanbck,medbck,stdbck,minbck,maxbck,meanvar,medvar,backmag,n_stars,mag_95q,m_chi2d,spatq50,spatq75,spatq90,spatq95,ast_nsrc,ast_arms,ast_errm,rp_mrms,rp_mmrms,pixscale,rotat,gm_ratex,gm_stdx,gm_ratey,gm_stdy,crpix1,crpix2,crval1,crval2,cd1_1,cd1_2,cd2_1,cd2_2,pv1_0,pv1_1,pv1_2,pv1_4,pv1_5,pv1_6,pv1_7,pv1_8,pv1_9,pv1_10,pv2_0,pv2_1,pv2_2,pv2_4,pv2_5,pv2_6,pv2_7,pv2_8,pv2_9,pv2_10,fwhe,fwhm,psf_fitn,psf_fita,psf_fitb,psf_fitt,psf_nst,psf_pkr,psf_dpk,psf_err,psf_s2,psf_af_1,psf_af_2,psf_af_3,psf_af_4,med_x2,med_y2,med_xy,snchi2,quicfwhm,ph_zp,ph_col1,ph_medc,ph_rms,ph_nsrc,pt_zp,pt_fnorm,limmag,limmager,pt_rms,pt_arms,pt_chi2,pt_dof,pt_ncali,pt_dzpab,pt_cta,pt_cta2,pt_cta3,pt_cta4,pt_ctae,pt_1_v1,pt_2_v1,pt_2_v2,pt_3_v1,pt_3_v2,pt_3_v3,pt_4_v1,pt_4_v2,pt_5_v1,pt_5_v2,pt_5_v3,pt_6_v1,pt_6_v2,pt_6_v3,pt_6_v4,pt_7_v1,pt_7_v2,pt_7_v3,pt_7_v4,pt_7_v5,pt_7_v6,pt_7_v7,pt_7_v8,pt_7_v9,pt_8_v1,pt_8_v2,pt_8_v3,pt_8_v4,pt_p_v1,pt_p_v2,pt_p_v3,pt_p_v4,apc0_a1,apcx_a1,apcy_a1,apcxy_a1,apcc_a1,apcce_a1,apc0_a2,apcx_a2,apcy_a2,apcxy_a2,apcc_a2,apcce_a2,apc0_a4,apcx_a4,apcy_a4,apcxy_a4,apcc_a4,apcce_a4,apc0_ps,apcx_ps,apcy_ps,apcxy_ps,apcc_ps,apcce_ps,apcor_nQuestions for @chentishler:
bloom_filteronupix_par(13% of the blocks for one partition). The alternative is a projection, a second copy of the table sorted by position,SELECT * ORDER BY (upix_par, id_coadd), which reads about one block per partition and doubles the size of the table (a few GB per year).(nodenumb, mountnum, camnum, id_coadd)would make it read only that camera, again at twice the size. Is it worth it?bloom_filteronLowCardinality(String)columns and projections; both work on 26.6 (euclid).All reactions