Implementing limited-access Portal instance for HEC water utilities

[tslawecki]

@kknee indicated the best way to control access to sensitive datasets in netCDF (e.g. model results for time of travel and/or dilution that may show the location of a water intake) is

1.Host the files on a separate TDS that has access control (at the time we found no way to control access to some datasets but not all on a single TDS)

2.Create a separate instance of OceansMap with a general login.

Putting this in as an issue for discussion on next coordination call (tentatively 2/2)

[kkoch]

@ColinNBrooks Status? what next?

[ColinNBrooks]

I believe I need your input on getting this working, starting with the example data set that Amanda shared with her email of April 19th, where she said:

"I'm attaching a couple of example average travel time products for Belle Isle swimming area (BP, can be public) and the Grosse Pointe Farms water treatment intake (GP, would need to be restricted access) in case they're helpful with deciding how to include them in the portal.
A metadata form for these products is also attached.

Thanks,

Amanda"

Can we move ahead with this starting example?

[kkoch]

I have the metadata record in hand and will try to get it into GN today or tomorrow.

.nc files sent via above email (not posted here due to confidentiality issues).

[ColinNBrooks]

I'm trying to find out when we might present this to the SEMCOG water treatment plants group. They asked for June 21st but that's the last day of the IAGLR conference. I've offered to present it in person at the July 26th meeting of the SEMCOG water treatment plant meeting, but we may also do a webinar sometime in June to help inform people.

[beckypearson]

Colin, I am following this issue. I think most of us will be at the IAGLR conference all that week as well. I more than happy to help with the webinar and/or inperson meeting, July 26.

[ColinNBrooks]

Thanks Becky. I'm hoping to settle the date for a potential June webinar so we know when this has to be up & running by. I think that would be a good opportunity for feedback before we have a final presentation in July.

[tslawecki]

@Bobfrat, @kknee , @cheryldmorse ... I ran into some issues with my test AWS server, so I've temporarily placed only the (public) Belle Isle swimming area average time netcdf on the regular GLOS TDS server, available at

http://tds.glos.us/thredds/tad.html?dataset=TAD

One thing I noticed - coordinates are in transverse mercator. Will this work or do we need to have MTRI reproject to geographic?

[Bobfrat]

Thanks Tad, projecting on geographic coordinates will make our lives much easier.

Also, any thoughts on a banner for this portal? Should it be different from the myglos banner?

[ColinNBrooks]

HECWFS Water Intake Spill Scenarios?

[tslawecki]

@ColinNBrooks - I talked with Kelly, Cheryl and Bob this morning and they think 6/13 is feasible. There are three immediate (small?) things:

• Deliver in a different projection (just saw your e-mail)
• Confirm title to display (and your comment)
• Discuss what "standard" GLOS datasets should be available in tool, and whether catalog is needed

[tslawecki]

I've placed the current (UTM) Belle Isle average time netCDF on an AWS t2.micro instance, available at http://34.219.157.82:8080/thredds/catalog/HECWFS-All/catalog.html?dataset=HECWFS-collection/BP-avgt.nc

I will lock down access to the 64.9.201.64/28 block (GLOS server stack) and LimnoTech, RPS, and the Oceansmap dev server (52.1.143.106) after addressing IP addresses with access overall (Issue 222).

@ColinNBrooks - send me any IPs at MTRI you think should have access too ...

[ColinNBrooks]

I'm getting our MTRI address range to accept.

[ColinNBrooks]

Would you like the rest of the data (public beach spill modeling data & restricted water intake spill modeling data) at this point as well? Since June 13th is fairly close I thought it might make sense to share it all.

[tslawecki]

Yes, but in geographic rather than UTM.

[kkoch]

Metadata record > http://data.glos.us/metadata/srv/eng/main.home?uuid=ca2ce240-7da9-489c-80fa-b5326dc28e91

[ColinNBrooks]

Here's what our IT head said:
For the main network have them add 35.63.66.0/23

(I assume that means 35.63.66.0 to 35.63.66.23)

[tslawecki]

@ColinNBrooks, I've uploaded the WGS84 versions of beach and plant maps, but exposed only the beach maps pending IP lockdown ... see http://34.219.157.82:8080/thredds/catalog/HECWFS-All/catalog.html.

[ColinNBrooks]

Cool!

[Bobfrat]

@tslawecki The WMS layers for this source are WMS version 1.3.0 which we need the latest version of OWSLib to parse in our data-catalog which has the potential of breaking during the GeoNetwork parsing. Can we safely make the upgrade? I know @cheryldmorse has been tracking down the problematic records.

[Bobfrat]

@tslawecki nevermind, we can use WMS 1.1.1 for these layers

[tslawecki]

@Bobfrat - Good, because my answer was along the lines of

I ... hope so?

We seem to have addressed the missing "uuidref" problems, but Cheryl found a different record where there is a missing "uuid" attribute we need to address. See latest from #194

[Bobfrat]

@tslawecki just verifying that the preferred WMS endpoint for these datasets is the WMS link from TDS. For example:

http://34.219.157.82:8080/thredds/wms/HECWFS-All/WY-prob.nc?service=WMS&version=1.3.0&request=GetCapabilities

[Bobfrat]

@tslawecki @ColinNBrooks, assuming the answer to the previous is yes, is it possible to aggregate the model variables (prob, mint, maxt, avet) into a single file (NCML?) so that the WMS endpoints contain all the layers in a single GetCapabilities request?

Without that it's not trivial for me to get all the layers into a single TOC entry like we do for the MTRI layers per lake:

Regarding metadata, I'm having a hard time figuring out what the file prefixes stand for (WY, VP, NB, MC, etc). It may be a good idea to move the Metadata variable attributes into the Global attributes to move closer in line with the ACDD/CF standards.

[ColinNBrooks]

I'm not sure so I'm asking Reid Sawtell ( rwsawtel@mtu.edu ) who helped set this up

[kknee]

@ColinNBrooks I think the metadata request that @Bobfrat mentioned would fall to MTRI, but the aggregation could be done on the TDS. @tslawecki can someone on LT tackle or do you want us to (I think we would need access to the server)?

[tslawecki]

@kknee - It's going to be fastest for RPS to do the aggregation. E-mailing you encrypted key.

[ColinNBrooks]

Water intake spill modeling locations (restricted access data):
IT = Ira Township
NB = New Baltimore
MC = Mount Clemens
GP = Grosse Pointe Farms - Highland Park
BP = Detroit - Belle Isle
FI = Detroit - Fighting Island
WY = Wyandotte

Beach spill modeling locations (publicly available data):
BI = Burke Park
VP = Veterans Park
MB = Metro Beach

(Yes, the abbrevations for Belle Isle = BP and Burke Park = BI are confusing, but those are right)

I'm working with Reid to try and get you updated combined data with metadata, but just in case that doesn't get to you in time, I wanted you to have this in case we need to use this to name layers clearly for Wednesday.

[Bobfrat]

@tslawecki we cant reach the THREDDS server this am. Can you have a look?

What do you have for available resources on that machine?

[tslawecki]

I can't get in either. Checking. It's a t2.micro instance, but basic monitoring doesn't look out of line ... [cid:122ff3ba-0db0-4adc-920f-1cb2c602bcbd]

…

________________________________ From: Bob Fratantonio <notifications@github.com> Sent: Monday, June 11, 2018 8:58 AM To: glos/myglos Cc: Tad Slawecki; Mention Subject: Re: [glos/myglos] Implementing limited-access Portal instance for HEC water utilities (#190) @tslawecki<https://github.com/tslawecki> we cant reach the THREDDS server this am. Can you have a look? What do you have for available resources on that machine? - You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#190 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AE-3PjEkwITB8boKgj-joHZ5_KAaaM6Aks5t7mlggaJpZM4Rt3Fz>.

[tslawecki]

@Bobfrat, seems to be fine after a reboot. LMK if you think it would be prudent to move to a larger instance (t2.medium?).

[Bobfrat]

How's the CPU credit balance look over the last few days?

[tslawecki]

Pegged at 145 or so.

[Bobfrat]

Ok that seems fine.

[ColinNBrooks]

Updated data - combined into one set per site, and with correct title in metadata.