Skip to content

Architecture 3

Jump to bottom
Nathan Potter edited this page Feb 2, 2017 · 16 revisions

Hyrax Running All CloudyDap Architectures

The code for Architecture 3 is now up and running. As I was writing A3 I realized that the code was (modulo a wee touch of parser mods) the same as the A2 code. And I realized that the only difference was really in the inputs: In A2 I added a top level XML attribute, xml:base, to the dmr++ files that identified the location of the (in the A2 case) HDF5 file against which the range-gets would be made, as described by each h4:byteStream content. In A3 each h4:byteStream is associated with a different S3 resource, so I add an href element to every h4:byteStream that identifies either a locally held resource or an HTTP accessible resource holding the h4:byteStream's data content. It was easy to tweak my dmr++ modification scripts to apply the A2 use-case using the href element addition to the h4:byteStream elements - the difference being that in A2 all the href attributes point to the same resource and the offset and nBytes attributes are used to locate the specific content required. In A3 all of the href attributes point to (mostly) different resources, the offset value is set to 0, and nBytes remains the same (and in theory should be same as the size of the referenced object).

Why is the A2 use-case the "same" code as before? Because under the hood in the C++ code the H4ByteStream objects used by the implementation always had a data_url. In the code we originally ran for A2, the parser collected the top level xml:base attribute and propagated it into each instance of H4ByteStream. In the new code the parser collects each h4:byteStream/@href and propagates it into the associated instance of H4ByteStream. After that, All Of The Downstream Processing Is The Same!

#Access

On the new test server I have two sets of dmr++ files, one for A2 and one for A3. Also, this server is running the S3 Gateway code which means the it does A1 too.

This server is running the code to process ALL of our tests for all of our architectures.

The fixed IP is here: http://52.20.142.186:8080/opendap/

And I have added it to our DNS service as cloudydap.opendap.org

Version Access URL
Architecture 1 http://cloudydap.opendap.org:8080/s3/cloudydap/
Architecture 2 http://cloudydap.opendap.org:8080/opendap/arch2/
Architecture 3 http://cloudydap.opendap.org:8080/opendap/arch3/

Shell Scripts For Use Cases

I branched the cloudydap github project for architecture 3:

https://github.com/OPENDAP/cloudydap/tree/arch3

And started adapting the content to work with the new A3 service stack that supports all three architectures.

Aggregation Emulator Scripts

I have altered the aggEmulator scripts as follows:

I created new versions of the aggregation emulator scripts, one for each architecture:

-rwxr-xr-x@ 1 ndp  staff       642 Jan 30 17:03 arch1_aggTestAirs
-rwxr-xr-x@ 1 ndp  staff       658 Jan 30 16:55 arch1_aggTestMerra2
-rwxr-xr-x@ 1 ndp  staff       644 Jan 30 16:49 arch2_aggTestAirs
-rwxr-xr-x@ 1 ndp  staff       659 Jan 30 16:49 arch2_aggTestMerra2
-rwxr-xr-x@ 1 ndp  staff       644 Jan 30 16:49 arch3_aggTestAirs
-rwxr-xr-x@ 1 ndp  staff       659 Jan 30 16:49 arch3_aggTestMerra2

Each of these expects a single command line parameter, the use-case ID. Everything else is hardwired, and we could probably hard code that bit too.

Usage Example:

./arch2_aggTestMerra2 UC6

Use-Case Scripts

The scripts in cloudydap/shell/use-cases will need additional work to map them into this new service stack.

TODO

  1. Adapt the shell scripts and other material in the cloudydap/shell/use-cases directory to utilize this EC2 system for all use-cases and architectures.
  2. Remove all unused material (scripts, files, bes command files, etc) from cloudydap/shell/use-cases. (Could be copied to an as yet to be created subdir named "unused" for example)
  3. In the use-cases dir we should have a set of scripts that run each use-case on each architecture. In addition, we should have three "master" scripts (one for each architecture) that launch all the use-cases for that particular architecture.
  4. My lack if familiarity with contents of cloudydap/shell/use-cases may mean that one or more of the above items is effectively a NO-OP.

Clone this wiki locally