New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
To avoid latency disable uniq sorting by default and made it optional #2898
To avoid latency disable uniq sorting by default and made it optional #2898
Conversation
A new Pull Request was created by @vkuznet (Valentin Kuznetsov) for branch IB/CMSSW_9_1_X/gcc530. @cmsbuild, @smuzaffar, @iahmad-khan, @davidlange6 can you please review it and eventually sign? Thanks. |
please test |
@smuzaffar, Shahzad could you please trigger tests build, I submit PR early last week. |
please also update dasgoclient.spec |
Pull request #2898 was updated. |
@cmsbuild, please test |
The tests are being triggered in jenkins. |
Comparison job queued. |
I found that recent change to use unique results may significantly hurt performance for large queries, such as
file dataset=/Neutrino_E-10_gun/RunIISpring15PrePremix-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v2-v2/GEN-SIM-DIGI-RAW
which returns 125K+ LFNs. Sorting of such list will take a minute and not really necessary. While for other DAS queries, such as site for given dataset, it is required since users want to see final set of sites, while results shows all possible replicas. As compromise I made unique flag and made it optional. By default sorting/uniq is disabled to provide ability to look-up results as fast as possible.