New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cropped Windowed GeoTiff Reading #1559

Merged
merged 8 commits into from Sep 23, 2016

Conversation

Projects
None yet
2 participants
@jbouffard
Contributor

jbouffard commented Jun 27, 2016

This pull request adds the following:

  1. A new implementation of CompressedBytes called SegmentBytes that allows for reading in the GeoTiff's segments into either a ByteBuffer or an Array[Array[Byte]].
  2. GeoTiffs can now be read in via a stream.
  3. GeoTiffs now have their own crop method which can be used in conjunction with the stream reading.
  4. Reading in a windowed GeoTiff given an Extent.
import monocle.syntax.apply._
import spire.syntax.cfor._
class CompressedBytesArray(compressedBytes: Array[Array[Byte]]) extends CompressedBytes {

This comment has been minimized.

@lossyrob

lossyrob Jun 27, 2016

Member

Naming convention goes from left being the most specific, right being the most general.
Not everything follows this convention in the base library, but we should follow it consistently in this library (and I think we generally do with a couple of exceptions perhaps).

ArrayCompressedBytes
ByteBufferCompressedBytes

would fit better.

Also perhaps a different name would be more descriptive.

SegmentBytes
ArraySegmentBytes
BufferSegmentBytes

Thoughts?

This comment has been minimized.

@jbouffard

jbouffard Jun 27, 2016

Contributor

I agree, those names were just place holders until I could think of something better. I like the names you proposed, and will rename the files and classes with their respective new names.

This comment has been minimized.

@jbouffard

jbouffard Jun 27, 2016

Contributor

Sorry about that, I had begun finishing up the unit tests, but I felt that I was late to make a PR so I just added my code without the tests. I'll be adding the tests soon.

This comment has been minimized.

@lossyrob

lossyrob Jun 27, 2016

Member

No worries, just figured I'd mention it explicitly, I know this is a WIP PR :)

@lossyrob

This comment has been minimized.

Member

lossyrob commented Jun 27, 2016

Should have unit tests around the implementations.

@jbouffard jbouffard force-pushed the jbouffard:windowedgeotiff branch 2 times, most recently from 87e8c5d to a7becd1 Jun 29, 2016

@jbouffard jbouffard force-pushed the jbouffard:windowedgeotiff branch from a7becd1 to bf26d45 Jul 6, 2016

@jbouffard jbouffard force-pushed the jbouffard:windowedgeotiff branch from bf26d45 to 4fdf9f5 Jul 18, 2016

@jbouffard jbouffard force-pushed the jbouffard:windowedgeotiff branch from cb81ad7 to b5b1c92 Jul 26, 2016

@jbouffard jbouffard force-pushed the jbouffard:windowedgeotiff branch from b5b1c92 to 0deccb4 Aug 11, 2016

@echeipesh echeipesh changed the title from [WIP] Added CompressedBytes abstraction to [WIP] Windowed GeoTiff Reading Aug 22, 2016

jbouffard added some commits Jun 27, 2016

Added CompressedBytes abstraction
Renamed the CompressedBytes abstractions

Renamed for real

Began writing tests

Added unit tests for ArraySegmentBytes

Made some changes to ArraySegmentBytes

Added BufferSegmentCountsTesting

Made changes to BufferSegmentBytes

Implemented SegmentBytes in the GeoTiff directory

Added map to SegmentBytes and made it work with Writer

Modified Boilerplate

Changed some of the testing

Added some more testing

Made chnages

Refactored SegmentBytes

Added the ability to read in windowed GeoTiffs to the implementations of GeoTiff

Refactored toArrayTile

Further edits to toArrayTile

Corrected toArrayTile

Made modifications

Implemented the WindowedGeoTiff class in the code

Refactored some more

Added most of the new Mutables

Added windowed reading

aslfkjsd

Reimplemented the modified mutables

Yet even more refactoring

Improved testing

Added documentation for most things

Removed CompressedBytes* files

Renamed test

Removed BufferSegmentBytes

Improved testing

made more changes to the tile files.

Modified implementation

Made more changes to the tests

Changed how rows and cols are found
Implemented the cropped reading for GeoTiffs
Made change to Float64

Improved testing to meet new implementation

Bit complete

reformatted tests

Added a note on Spec

Made a few formatting changes

Made changes to sbt for local publishing

Made improvements to Windowed Readers speed

fixed build.sbt

Made striped reading faster for all

made some improvements to the reader

Made the test a little better

Corrected error with the tiled reader7

Updated tests

Solved the issue for real this time

working on bug fixes
Renamed the functions and changed their implementation
made changes to the striped reader

Tiles actually work now.

They work for real now

Implementation changed

Implementation improve and added to all types

Cleaned up CroppedGeoTiff

Cleaned up code and made a few more changes
Improved speed for some bandtypes
Tile reading works without checking now.

Changed Int16 and UInt16

Changed formatting

Made a few more changes

Fixed Tile Reading issue

Implemented the correct API

Changed how the reading is done in single and mulitband files

Made changes to the test

Made fromInputStream implicit

made some changes to how the buffer is read

Made the testing a little better

@jbouffard jbouffard force-pushed the jbouffard:windowedgeotiff branch 2 times, most recently from 1e4e407 to aeaf095 Aug 28, 2016

Cleaned up the code further
Removed S3 stuff for good

Edited code some more

Improved SegmentBytesSpec

Edited CroppedWindowedGeoTiffSpec

Addd removed file

@jbouffard jbouffard force-pushed the jbouffard:windowedgeotiff branch from aeaf095 to 53c3e38 Aug 29, 2016

@jbouffard jbouffard changed the title from [WIP] Windowed GeoTiff Reading to Cropped Windowed GeoTiff Reading Aug 29, 2016

@lossyrob

This comment has been minimized.

Member

lossyrob commented Aug 29, 2016

@jbouffard can you add a description of the feature in the PR description? Something that if people were to find this PR after the fact, they could understand what the feature being merged in is all about.

@jbouffard

This comment has been minimized.

Contributor

jbouffard commented Aug 29, 2016

@lossyrob Yeah, I just added one.

This was referenced Aug 29, 2016

@jbouffard

This comment has been minimized.

Contributor

jbouffard commented Aug 30, 2016

@lossyrob I looked at where the checked failed, and I don't think it had to do with my code. This is the error message that caused it to fail:

The future returned an exception of type: java.sql.BatchUpdateException, with message: Batch entry 0 insert into "cities" ("name","geom") values ('Allentown,PA',?) was aborted. Call getNextException to see the cause.. (PostgisSpec.scala:112)

Files are now read through streaming
Made streaming better

@jbouffard jbouffard force-pushed the jbouffard:windowedgeotiff branch from 785f3e7 to b1da186 Aug 31, 2016

@jbouffard jbouffard referenced this pull request Aug 31, 2016

Merged

Stream Reading GeoTiffs From S3 #1617

3 of 3 tasks complete
@lossyrob

This comment has been minimized.

Member

lossyrob commented Sep 23, 2016

💯

@lossyrob lossyrob merged commit 6bd3052 into locationtech:master Sep 23, 2016

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

@lossyrob lossyrob added this to the 1.0 milestone Oct 18, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment