Thunder integration with OCP #130
Conversation
…ta from the OCP database by specifying the bucket name, resolution, minBound, maxBound and time range.
…t of URL's which fetahs npz data from OCP.
…then fetches sample code for testing.
… the data in xyz order. Tests will pass even if OCP is down.
|
Fixed the bug on our end. All tests now pass for all servers on our end. |
|
@kunallillaney do you mind merging in the latest changes from master? I now think the problem is that the test output with your changes became longer that Travis's maximum length, but some tooling I just did simplifies the test output dramatically, and should fix it if that's the problem. |
|
@freeman-lab Done and all tests pass. And the test output is now much better and readable. Once your are done with the series changes we can also work on the seriesloader for OCP. Thanks. |
|
|
||
| return data | ||
|
|
||
| rdd = self.sc.parallelize (enumerate(urlList), len(urlList)).map(lambda (k, v): (k, read(v))) |
There was a problem hiding this comment.
remove space after parallelize
|
@kunallillaney Thanks for the updates. I left a few comments. There are many deviations from PEP conventions for spacing here, can you open in PyCharm and try to fix them? Also, after looking at it more, I would rather move the top-level Can you try making that change? |
|
@freeman-lab Did fix all those(and more) PEP errors in PyCharm. Moved OCP out in the context file. Let me know all of this looks good. |
|
@freeman-lab Is there any more TODO's on our end? |
| @@ -133,6 +133,107 @@ def toArray(idxAndBuf): | |||
| newDims = tuple(list(dims[:-1]) + [nplanes]) if nplanes else dims | |||
| return Images(readerRdd.flatMap(toArray), nrecords=nrecords, dims=newDims, dtype=dtype) | |||
|
|
|||
| def fromOCP (self, bucketName, resolution, serverName='ocp.me', startIdx=None, stopIdx=None, minBound=None, | |||
| maxBound=None): | |||
| """Sets up a new Image object with data to read from OCP | |||
|
Just left a couple final requests, and can you merge in from master to figure out the merge conflict? After that we can merge this in! Thanks. |
|
@freeman-lab Done and pushed. |
|
Great! LGTM, thanks for the contribution @kunallillaney ! |
Thunder integration with OCP
|
Thanks. |
Hey Jeremy,
I have merged the latest branch of thunder and documented my function. In addition, the tests are also fixed. They will not fail if OCP is down.