Duplicate rows in group source downloads

[bfhealy]

When downloading group sources from Fritz using the frontend or api, the result sometimes contains duplicate rows (although always the correct number of total rows, so some sources are missing). I have not encountered this problem for smaller groups (<~ 1000 sources), but for larger groups I get a variable number of duplicates each time I download. Some additional info:

• There are no duplicates on any single page; they instead appear on separate pages.
• The number of duplicates increases for a large ratio of total sources to number per page.
• I have been unable to replicate this behavior on my local SkyPortal instance, including in scenarios where sources are members of multiple groups or other sources are in the process of being posted.

It is possible that modifying this call of grab_query_results:

skyportal/skyportal/handlers/api/source.py

Line 1026 in 91a293f

query_results = grab_query_results(

by setting use_cache=True, see

skyportal/skyportal/handlers/api/candidate.py

Line 1234 in 03f7c79

def grab_query_results(

will solve the problem. We'd also have to decide on how long to maintain the cache.

[bfhealy]

I've tested setting use_cache=True as described above in my local Skyportal instance, and group source downloads (frontend and api) still work as expected. Is there any other functionality that may be adversely affected by setting this keyword?

[mcoughlin]

@guynir42 What do you think of trying this and seeing how it goes this week?

[guynir42]

You mean adding use_cache=True to the download CSV option?

[mcoughlin]

@guynir42 yes.

[bfhealy]

Unfortunately PR #3569 has not resolved the duplicate row issue with Fritz. I am still unable to replicate the issue locally.

[bfhealy]

Testing on a group of ~10,000 sources in preview did not produce any duplicates. It seems this issue is specific to fritz.science.

[bfhealy]

@mcoughlin @guynir42 In the last several weeks, my downloads of group sources on Fritz have produced no duplicates. Not sure what changed for the better, but I'll close this issue for now.