Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When multiple Reaper instances are running on the same node, the RSE configuration somehow gets mixed #4069

Closed
dchristidis opened this issue Oct 13, 2020 · 0 comments · Fixed by #4552
Assignees
Milestone

Comments

@dchristidis
Copy link
Contributor

For example, on a VM where a Reaper instance is configured like so with Supervisor:

[program:reaper]
command=/usr/bin/rucio-reaper2 --threads 50  --include-rses "(type=CALIBDISK|type=DATADISK|type=GROUPDISK|type=LOCALGROUPDISK|type=SCRATCHDISK|type=TEST|CERN-PROD_TZDISK)\quarantinereaper=True"
…

Starting an instance on a particular RSE:

# rucio-reaper2 --include-rses CERN-PROD_TEST-CTA --run-once
2020-10-13 14:17:58,861	21869	INFO	main: starting processes
2020-10-13 14:17:58,891	21869	INFO	Reaper: This instance will work on RSEs: AUSTRALIA-ATLAS_DATADISK, AUSTRALIA-ATLAS_LOCALGROUPDISK, AUSTRALIA-ATLAS_SCRATCHDISK, AUSTRALIA-ATLAS_T2ATLASLOCALGROUPDISK, BEIJING-LCG2_DATADISK, BEIJING-LCG2_LOCALGROUPDISK, BNL-OSG2_DATADISK, BNL-OSG2_DET-SLHC, BNL-OSG2_LOCALGROUPDISK, BNL-OSG2_PHYS-HI, BNL-OSG2_SCRATCHDISK, BNLHPC_SCRATCHDISK, CA-SFU-T2_DATADISK, CA-SFU-T2_LOCALGROUPDISK, CA-SFU-T2_SCRATCHDISK, CA-TRIUMF-DYNAFED_DATADISK, CA-TRIUMF-DYNAFED_SCRATCHDISK, CA-UVIC-CLOUD_DATADISK, CA-UVIC-CLOUD_SCRATCHDISK, CA-VICTORIA-WESTGRID-T2_DATADISK, CA-VICTORIA-WESTGRID-T2_LOCALGROUPDISK, CA-VICTORIA-WESTGRID-T2_SCRATCHDISK, CA-WATERLOO-T2_DATADISK, CA-WATERLOO-T2_LOCALGROUPDISK, CA-WATERLOO-T2_SCRATCHDISK, CERN-EXTENSION_SCRATCHDISK, CERN-PROD_DATADISK, CERN-PROD_DATAPREP, CERN-PROD_DET-FWD, CERN-PROD_DET-IBL, CERN-PROD_DET-INDET, CERN-PROD_DET-LARG, CERN-PROD_DET-MUON, CERN-PROD_DET-SLHC, CERN-PROD_DET-TILE, CERN-PROD_LOCALGROUPDISK, CERN-PROD_PERF-EGAMMA, CERN-PROD_PERF-FLAVTAG, CERN-PROD_PERF-IDTRACKING, CERN-PROD_PERF-JETS, CERN-PROD_PERF-MUONS, CERN-PROD_PERF-TAU, CERN-PROD_PHYS-BEAUTY, CERN-PROD_PHYS-EXOTICS, CERN-PROD_PHYS-GENER, CERN-PROD_PHYS-HDBS, CERN-PROD_PHYS-HI, CERN-PROD_PHYS-HIGGS, CERN-PROD_PHYS-SM, CERN-PROD_PHYS-SUSY, CERN-PROD_PHYS-TOP, CERN-PROD_PPSTAPE, CERN-PROD_PROJ-SIT, CERN-PROD_SCRATCHDISK, CERN-PROD_SOFT-SIMUL, CERN-PROD_TEST-CTA, CERN-PROD_TRIG-DAQ, CERN-PROD_TRIG-HLT, CERN-PROD_TZDISK, CSCS-LCG2_DATADISK, CSCS-LCG2_LOCALGROUPDISK, CSCS-LCG2_SCRATCHDISK, CYFRONET-LCG2_DATADISK, CYFRONET-LCG2_LOCALGROUPDISK, CYFRONET-LCG2_PHYS-HI, CYFRONET-LCG2_SCRATCHDISK, DESY-HH_DATADISK, DESY-HH_LOCALGROUPDISK, DESY-HH_SCRATCHDISK, DESY-ZN_DATADISK, DESY-ZN_LOCALGROUPDISK, DESY-ZN_PHYS-TOP, DESY-ZN_SCRATCHDISK, EELA-UTFSM_DATADISK, EELA-UTFSM_LOCALGROUPDISK, EELA-UTFSM_SCRATCHDISK, FMPHI-UNIBA_DATADISK, FMPHI-UNIBA_LOCALGROUPDISK, FMPHI-UNIBA_SCRATCHDISK, FR-ALPES_DATADISK_DATALAKES, FR-ALPES_SCRATCHDISK_DATALAKES, FZK-LCG2_DATADISK, FZK-LCG2_PERF-EGAMMA, FZK-LCG2_PERF-IDTRACKING, FZK-LCG2_PERF-TAU, FZK-LCG2_SCRATCHDISK, GOEGRID_DATADISK, GOEGRID_DET-INDET, GOEGRID_LOCALGROUPDISK, GOEGRID_SCRATCHDISK, GRIF-IRFU_DATADISK, GRIF-IRFU_LOCALGROUPDISK, GRIF-IRFU_SCRATCHDISK, GRIF-LAL_DATADISK, GRIF-LAL_LOCALGROUPDISK, GRIF-LAL_SCRATCHDISK, GRIF-LPNHE_DATADISK, GRIF-LPNHE_LOCALGROUPDISK, GRIF-LPNHE_SCRATCHDISK, HK-LCG2_DATADISK, HK-LCG2_SCRATCHDISK, IEPSAS-KOSICE_DATADISK, IEPSAS-KOSICE_LOCALGROUPDISK, IEPSAS-KOSICE_SCRATCHDISK, IFAE_DATADISK, IFAE_LOCALGROUPDISK, IFAE_SCRATCHDISK, IFIC-LCG2_CALIBDISK, IFIC-LCG2_DATADISK, IFIC-LCG2_LOCALGROUPDISK, IFIC-LCG2_SCRATCHDISK, IL-TAU-HEP_DATADISK, IN2P3-CC_DATADISK, IN2P3-CC_DATALAKE_DATADISK, IN2P3-CC_LOCALGROUPDISK, IN2P3-CC_LOCALGROUPDISK-MW, IN2P3-CC_PERF-FLAVTAG, IN2P3-CC_PHYS-HIGGS, IN2P3-CC_PHYS-SM, IN2P3-CC_SCRATCHDISK, IN2P3-CPPM_DATADISK, IN2P3-CPPM_LOCALGROUPDISK, IN2P3-CPPM_SCRATCHDISK, IN2P3-LAPP_DATADISK, IN2P3-LAPP_LOCALGROUPDISK, IN2P3-LAPP_SCRATCHDISK, IN2P3-LPC_DATADISK, IN2P3-LPC_LOCALGROUPDISK, IN2P3-LPC_SCRATCHDISK, IN2P3-LPSC_DATADISK, IN2P3-LPSC_LOCALGROUPDISK, IN2P3-LPSC_SCRATCHDISK, INFN-BOLOGNA-T3_LOCALGROUPDISK, INFN-COSENZA_DATADISK, INFN-COSENZA_LOCALGROUPDISK, INFN-COSENZA_SCRATCHDISK, INFN-FRASCATI_DATADISK, INFN-FRASCATI_LOCALGROUPDISK, INFN-FRASCATI_SCRATCHDISK, INFN-GENOVA_DATADISK, INFN-GENOVA_LOCALGROUPDISK, INFN-GENOVA_SCRATCHDISK, INFN-LECCE_LOCALGROUPDISK, INFN-LECCE_SCRATCHDISK, INFN-MILANO-ATLASC_DATADISK, INFN-MILANO-ATLASC_LOCALGROUPDISK, INFN-MILANO-ATLASC_SCRATCHDISK, INFN-NAPOLI-ATLAS_CALIBDISK, INFN-NAPOLI-ATLAS_DATADISK, INFN-NAPOLI-ATLAS_LOCALGROUPDISK, INFN-NAPOLI-ATLAS_SCRATCHDISK, INFN-NAPOLI-ATLAS_TRIG-DAQ, INFN-ROMA1_CALIBDISK, INFN-ROMA1_DATADISK, INFN-ROMA1_LOCALGROUPDISK, INFN-ROMA1_SCRATCHDISK, INFN-ROMA2_LOCALGROUPDISK, INFN-ROMA3_DATADISK, INFN-ROMA3_LOCALGROUPDISK, INFN-ROMA3_SCRATCHDISK, INFN-T1_DATADISK, INFN-T1_SCRATCHDISK, INFN-TRIESTE_LOCALGROUPDISK, JINR-LCG2_DATADISK, JINR-LCG2_LOCALGROUPDISK, JINR-LCG2_SCRATCHDISK, LRZ-LMU_CALIBDISK, LRZ-LMU_DATADISK, LRZ-LMU_LOCALGROUPDISK, LRZ-LMU_SCRATCHDISK, MAINZ_LOCALGROUPDISK, MPPMU_CALIBDISK, MPPMU_DATADISK, MPPMU_LOCALGROUPDISK, MPPMU_PERF-JETS, MPPMU_PERF-MUONS, MPPMU_SCRATCHDISK, MWT2_DATADISK, MWT2_UC_LOCALGROUPDISK, MWT2_UC_SCRATCHDISK, NCG-INGRID-PT_DATADISK, NCG-INGRID-PT_LOCALGROUPDISK, NCG-INGRID-PT_SCRATCHDISK, NDGF-T1_DATADISK, NDGF-T1_DATADISK-DATAFJORD, NDGF-T1_PHYS-HIGGS, NDGF-T1_PHYS-SUSY, NDGF-T1_SCRATCHDISK, NDGF-T1_TRIG-DAQ, NDGF-T1_TRIG-L1CALO, NIKHEF_DATADISK, NIKHEF_LOCALGROUPDISK, NIKHEF_SCRATCHDISK, NO-NORGRID-T2_LOCALGROUPDISK, OLCF_SCRATCHDISK, PIC_DATADISK, PIC_SCRATCHDISK, PRAGUELCG2_DATADISK, PRAGUELCG2_LOCALGROUPDISK, PRAGUELCG2_PHYS-HI, PRAGUELCG2_SCRATCHDISK, PSNC_DATADISK, PSNC_SCRATCHDISK, RO-02-NIPNE_LOCALGROUPDISK, RO-07-NIPNE_DATADISK, RO-07-NIPNE_LOCALGROUPDISK, RO-07-NIPNE_SCRATCHDISK, RRC-KI-T1_DATADISK, RRC-KI-T1_LOCALGROUPDISK, RRC-KI-T1_SCRATCHDISK, RU-PROTVINO-IHEP_DATADISK, RU-PROTVINO-IHEP_LOCALGROUPDISK, RU-PROTVINO-IHEP_SCRATCHDISK, SARA-MATRIX_DATADISK, SARA-MATRIX_SCRATCHDISK, SE-SNIC-T2_LUND_LOCALGROUPDISK, SE-SNIC-T2_STHLM_LOCALGROUPDISK, SE-SNIC-T2_UPPSALA_LOCALGROUPDISK, SIGNET_LOCALGROUPDISK, TAIWAN-LCG2_CALIBDISK, TAIWAN-LCG2_DATADISK, TAIWAN-LCG2_LOCALGROUPDISK, TAIWAN-LCG2_SCRATCHDISK, TECHNION-HEP_DATADISK, TECHNION-HEP_LOCALGROUPDISK, TECHNION-HEP_SCRATCHDISK, TOKYO-LCG2_DATADISK, TOKYO-LCG2_DET-MUON, TOKYO-LCG2_LOCALGROUPDISK, TOKYO-LCG2_PERF-MUONS, TOKYO-LCG2_PHYS-EXOTICS, TOKYO-LCG2_SCRATCHDISK, TR-10-ULAKBIM_DATADISK, TR-10-ULAKBIM_LOCALGROUPDISK, TR-10-ULAKBIM_SCRATCHDISK, TRIUMF-LCG2_DATADISK, TRIUMF-LCG2_LOCALGROUPDISK, TRIUMF-LCG2_PERF-JETS, TRIUMF-LCG2_PERF-TAU, TRIUMF-LCG2_SCRATCHDISK, TUDRESDEN-ZIH_LOCALGROUPDISK, UAM-LCG2_DATADISK, UAM-LCG2_LOCALGROUPDISK, UAM-LCG2_SCRATCHDISK, UKI-LT2-BRUNEL_DATADISK, UKI-LT2-QMUL_DATADISK, UKI-LT2-QMUL_LOCALGROUPDISK, UKI-LT2-QMUL_SCRATCHDISK, UKI-LT2-RHUL_DATADISK, UKI-LT2-RHUL_LOCALGROUPDISK, UKI-LT2-RHUL_SCRATCHDISK, UKI-NORTHGRID-LANCS-HEP_DATADISK, UKI-NORTHGRID-LANCS-HEP_LOCALGROUPDISK, UKI-NORTHGRID-LANCS-HEP_PHYS-BEAUTY, UKI-NORTHGRID-LANCS-HEP_SCRATCHDISK, UKI-NORTHGRID-LIV-HEP_DATADISK, UKI-NORTHGRID-LIV-HEP_LOCALGROUPDISK, UKI-NORTHGRID-LIV-HEP_SCRATCHDISK, UKI-NORTHGRID-MAN-HEP_DATADISK, UKI-NORTHGRID-MAN-HEP_LOCALGROUPDISK, UKI-NORTHGRID-MAN-HEP_SCRATCHDISK, UKI-SCOTGRID-DURHAM_DATADISK, UKI-SCOTGRID-ECDF_DATADISK, UKI-SCOTGRID-ECDF_LOCALGROUPDISK, UKI-SCOTGRID-ECDF_SCRATCHDISK, UKI-SCOTGRID-GLASGOW_DATADISK, UKI-SCOTGRID-GLASGOW_LOCALGROUPDISK, UKI-SCOTGRID-GLASGOW_SCRATCHDISK, UKI-SOUTHGRID-OX-HEP_DATADISK, UKI-SOUTHGRID-OX-HEP_IPV6TEST, UKI-SOUTHGRID-OX-HEP_LOCALGROUPDISK, UKI-SOUTHGRID-OX-HEP_SCRATCHDISK, UKI-SOUTHGRID-RALPP_DATADISK, UKI-SOUTHGRID-RALPP_LOCALGROUPDISK, UKI-SOUTHGRID-RALPP_SCRATCHDISK, UNI-BONN_LOCALGROUPDISK, UNI-FREIBURG_DATADISK, UNI-FREIBURG_LOCALGROUPDISK, UNI-FREIBURG_SCRATCHDISK, UNI-SIEGEN-HEP_LOCALGROUPDISK, UNIBE-LHEP_DATADISK, UNIBE-LHEP_LOCALGROUPDISK, UNIBE-LHEP_SCRATCHDISK, UNICPH-NBI_LOCALGROUPDISK, USTC-T3_LOCALGROUPDISK, WEIZMANN-LCG2_LOCALGROUPDISK, WUPPERTALPROD_DATADISK, WUPPERTALPROD_LOCALGROUPDISK, WUPPERTALPROD_SCRATCHDISK, ZA-WITS-CORE_DATADISK, ZA-WITS-CORE_LOCALGROUPDISK, ru-PNPI_XCACHE_NODES
…
cserf added a commit to cserf/rucio that referenced this issue Apr 19, 2021
… the RSE configuration somehow gets mixed : Closes rucio#4069
bari12 added a commit that referenced this issue Apr 22, 2021
…nstances_are_running_on_the_same_node__the_RSE_configuration_somehow_gets_mixed

Reaper : When multiple Reaper instances are running on the same node, the RSE configuration somehow gets mixed : Closes #4069
bari12 pushed a commit that referenced this issue Apr 22, 2021
… the RSE configuration somehow gets mixed : Closes #4069
@bari12 bari12 added this to the 1.25.4 milestone Apr 22, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment