Filter Out Some Results from CUAHSI by Default

[rajadain]

To show only the most relevant results, we should hide certain elements (results such as NLDAS_FORA, NLDAS_NOAH) unless the user explicitly wants to show them. Because these gridded results are always up to date they show up at the top, but might not be the most relevant for most users.

@aufdenkampe / @emiliom to provide the final set of strings to use for filtering.

A simple checkbox for these items can be added here:

But there's a larger conversation to have (on another forum) about how we display global and tab-specific filters. /cc @jfrankl @ajrobbins

[ajrobbins]

I spoke to @jfrankl about this, and he's working on a design to accomodate a few catalog-specific filters including this one.

[emiliom]

I'm adding here services I've identified "manually" so far, to be filtered out (updated 8/21):

• NLDAS_FORA
• NLDAS_NOAH
• GLDAS_NOAH
• TRMM_3B42_7

Note that the simplest, initial filter we should focus on is the "gridded" services. The 3 services listed are all gridded services. The CUAHSI Data Services Help Center article "Gridded" Data Services in the HIS Central Catalog provides a brief description.

No action needed at this time. This is a reference for myself (and Anthony, I guess). We'll contact CUAHSI WDC for a definitive list of services or mechanism to filter them out more robustly.

[aufdenkampe]

Thanks for follow up on this!

[ajrobbins]

To be clear @emiliom, these services would be filtered out by the user in the UI? Or we would filter them out on the backend and never show them to the user?

[emiliom]

To be clear @emiliom, these services would be filtered out by the user in the UI? Or we would filter them out on the backend and never show them to the user?

My intention was the former. @aufdenkampe, do you agree? I think @rajadain described it well when he opened this issue: "A simple checkbox for these items can be added here". But maybe a refined feature description is that the request issued to CUAHSI WDC would most likely not include a filtering out mechanism; the filtering would be applied client side on the response. In that sense, the checkbox would serve to hide some result records based on service codes.

FYI, I've submitted a request to CUAHSI about this. I've also updated my running reference comment above

[ajrobbins]

Thanks for the confirmation, @emiliom. @jfrankl is working on design for a client-side filter.

[aufdenkampe]

I agree with @emiliom on the simple checkbox in UI approach, that filters out all the services we put in that category.

[emiliom]

Here's the response from CUAHSI WDC, from Martin Seul:

The functionality is implemented by specifying only the services you want to search, so we disregard a set of services that we manually specify currently these are these:

networkid :: networkname :: networktitle

• 187 :: NWS-WGRFC_Hourly_MPE :: NWS-WGRFC Hourly Multi-sensor Precipitation Estimates
• 189 :: NWS_WGRFC_Daily_MPE_Recent_Values :: NWS-WGRFC Daily Multi-sensor Precipitation Estimates Recent Values
• 226 :: LMRFC_Data :: NWS-LMRFC Hourly Multi Sensor Precipitation Estimates
• 262 :: GLDAS_NOAH :: GLDAS 3-hourly Noah Data
• 267 :: NLDAS_FORA :: NLDAS Hourly Primary Forcing Data
• 274 :: NLDAS_NOAH :: NLDAS Hourly NOAH Data
• 479 :: TRMM_3B42_7 :: TRMM and Others Rainfall Estimate Data (3-hourly 0.25 degree)

So we actually send a list of comma separated values as arguments in the request body see example below in this case we only search the gridded service:

xmin=-70&xmax=-62&ymin=40&ymax=48&conceptKeyword=&networkIDs=84%2C+187%2C+189%2C+226%2C+262%2C+267%2C+274%2C+479&beginDate=01%2F01%2F2010&endDate=01%2F01%2F2011.

Hope this helps. Please let me know if you need additional info.

[rajadain]

Hi @emiliom,

Looking at the sample request, it seems like the networkIDs=84, 187, 189, 226, 262, 267, 274, 479 parameter would return only the gridded services (which we want to exclude). To get the services we want, we can do one of three things:

1. Exclude certain networkIDs from the request. Not sure if the API supports this sort of negation.
2. Send networkIDs for all the services we want. In this case, we will need a full list of IDs to include.
3. Fetch all results, and hide the ones specified above.

If we expect heavy use of the checkbox filter, allowing the user to turn on the gridded results, option 3 would be the best user experience since the results would be instantaneous instead of waiting for a new API call. If, on the other hand, use of gridded results is rare, then we should go with either option 1 or option 2, which would speed up the initial request.

[emiliom]

Option 3 is the only one that's truly accessible to us and practical for our purposes. Let's go with it. On Aug 25, 2017 7:51 AM, "Terence Tuhinanshu" <notifications@github.com> wrote: Hi @emiliom <https://github.com/emiliom>, Looking at the sample request, it seems like the networkIDs=84, 187, 189, 226, 262, 267, 274, 479 parameter would return *only* the gridded services (which we want to exclude). To get the services we want, we can do one of three things: 1. Exclude certain networkIDs from the request. Not sure if the API supports this sort of negation. 2. Send networkIDs for all the services we want. In this case, we will need a full list of IDs to include. 3. Fetch all results, and hide the ones specified above. If we expect heavy use of the checkbox filter, allowing the user to turn on the gridded results, option 3 would be the best user experience since the results would be instantaneous instead of waiting for a new API call. If, on the other hand, use of gridded results is rare, then we should go with either option 1 or option 2, which would speed up the initial request. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2133 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAtUA3XmbATwNPMT-hqZD1aNaX79TjsCks5sbt9kgaJpZM4Oyss2> .

[aufdenkampe]

@rajadain, good points! Thanks for laying out the options.

I think we want to exclude the gridded services by default, such that a user would need to actively uncheck the "Exclude gridded services" box (or alternately, check the "Include gridded services" box). This checkbox might even be hidden in the default view (or even buried in user Settings).

Therefore, can we implement option 2 to speed performance? For large areas, these gridded services dominate the return, and therefore massively slow things down.

What is the implication to sending a web service request with a list of 92-7 = 85 networkid values?

[aufdenkampe]

@emiliom and I just gave different responses, but I suspect that he answered with option 3 primarily because of suspicions about practicality.

@rajadain, what is the practicality of option 2?
We would probably want to refresh our list of services every night or week, so that any new services that are registered will show up.

[aufdenkampe]

@rajadain, because WDC catalog responses are slow, we really want to prioritize response speed.

It will be rare for a user to want gridded services, and I'm fine with sending a second, much slower request, when the user choses that.

[rajadain]

That's good to know, @aufdenkampe. I think for the use case you describe, option 2 is best. I had started some work for option 3, but it can be changed to target this. Additionally, I will prioritize #1932 for the next sprint. Thanks for the feedback!

[ajrobbins]

(To be clear, #1932 is the work required to cache the list of services on a regular basis)