New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Write code to convert binary image files to full resolution geoTIFF + jpg preview #64

Closed
dlebauer opened this Issue Feb 15, 2016 · 23 comments

Comments

Projects
None yet
10 participants
@dlebauer
Member

dlebauer commented Feb 15, 2016

The field scanner sensors are exporting binary files (with .bin extension) along with .json files that with a field "Output Data format" that describes how the binary file was written, e.g. the FLIR camera files are "Output data format": "Bayer GR8 3296x2472", stereo cameras are"Output data format": "Bayer GR8 3296x2472",`

Here are the datasets: https://uofi.box.com/s/c9ngkwi3xvtfr0ei5zfzgf4jya0dmrlc

The Feb 9 folder has one sample from each of the sensors. The feb 12 folder has many images from the sensors (FLIR thermal IR, Stereo Vis, and Fluorescence).

Is there a general way to parse the meta-data files to find the format, and then convert the .bin files to something that can be read by computer programs? Are there existing BrownDog converters that we can use for these files?

  • @pless will write scripts that read information from metadata.json and convert these files to a lossless TIFF + jpg thumbnail along with a polygon describing the field of view.
    • scripts will take input and output folders as arguments. We will work
  • At March 14 meeting in St Louis we can work with him to integrate these into Clowder.

Update

  • We have octave / matlab code to convert PS2, NDVI, FLIR from Lemnatec in scripts/
  • next step: add extractors for these
@max-zilla

This comment has been minimized.

Show comment
Hide comment
@max-zilla

max-zilla Feb 15, 2016

Contributor

I don't have much experience here but I found this Quora page which suggests using an ImageJ plugin - here's a list of cameras the plugin supports (Find "Supported Cameras"). Perhaps a place to start if there isn't already a solution at hand.

Contributor

max-zilla commented Feb 15, 2016

I don't have much experience here but I found this Quora page which suggests using an ImageJ plugin - here's a list of cameras the plugin supports (Find "Supported Cameras"). Perhaps a place to start if there isn't already a solution at hand.

@nfahlgren

This comment has been minimized.

Show comment
Hide comment
@nfahlgren

nfahlgren Feb 15, 2016

Contributor

The code we have used for converting images in RAW format from our system is here: https://github.com/danforthcenter/PhenoFront/tree/master/src/main/java/src/ddpsc/results. The Bayer2Rgb.java is probably the most relevant for example.

Contributor

nfahlgren commented Feb 15, 2016

The code we have used for converting images in RAW format from our system is here: https://github.com/danforthcenter/PhenoFront/tree/master/src/main/java/src/ddpsc/results. The Bayer2Rgb.java is probably the most relevant for example.

@solmazhajmohammadi

This comment has been minimized.

Show comment
Hide comment
@solmazhajmohammadi

solmazhajmohammadi Feb 15, 2016

Currently I am using JSON-input parser MATLAB toolbox to parse the meta-data files to find the format. Then simply you can open the file and save it in specific filetype.

Currently I am using JSON-input parser MATLAB toolbox to parse the meta-data files to find the format. Then simply you can open the file and save it in specific filetype.

@yanliu-chn

This comment has been minimized.

Show comment
Hide comment
@yanliu-chn

yanliu-chn Feb 15, 2016

From what I know in general:
Droid from National Archives: http://digital-preservation.github.io/droid/ is a metadata identification and recording tool.

Commercial software https://www.safe.com/ provides conversion support for a lot of file formats, esp. GIS data.

From what I know in general:
Droid from National Archives: http://digital-preservation.github.io/droid/ is a metadata identification and recording tool.

Commercial software https://www.safe.com/ provides conversion support for a lot of file formats, esp. GIS data.

@dlebauer dlebauer assigned pless and unassigned solmazhajmohammadi Mar 3, 2016

@rachelshekar rachelshekar referenced this issue Mar 3, 2016

Closed

MAC demo preparation #61

0 of 3 tasks complete

@dlebauer dlebauer added the 1 - Ready label Mar 3, 2016

@dlebauer dlebauer changed the title from binary file converter to binary file converters Mar 17, 2016

@dlebauer dlebauer added sensor data and removed sensor data labels Mar 30, 2016

@dlebauer dlebauer changed the title from binary file converters to Write code to convert binary image files to full resolution geoTIFF + jpg preview Apr 18, 2016

@robkooper

This comment has been minimized.

Show comment
Hide comment
@robkooper

robkooper Apr 21, 2016

Member

One command that used to convert some of the raw data is:

convert -size 640x480 -depth 16 gray:input_ir.bin -evaluate multiply 16 -depth 8 output.png
Member

robkooper commented Apr 21, 2016

One command that used to convert some of the raw data is:

convert -size 640x480 -depth 16 gray:input_ir.bin -evaluate multiply 16 -depth 8 output.png
@dlebauer

This comment has been minimized.

Show comment
Hide comment
@dlebauer

dlebauer Apr 21, 2016

Member

@robkooper thanks. That looks like a good first step, additional steps for this issue include:

  1. creating geoTIFF.
    • presumably requires subsequent step with gdal or similar
    • also requires knowledge of image extent
  2. thumbnail generation:
    • Does Clowder already have a thumbnail extractor that we can use?
    • Or, what resolution or max pixel count do you recommend? to handle sensors with different dimensions, we can set max pixel count as convert input.png -resize 4096@ thumbnail.jpg according to imagemagik resize docs.

Would it be worth it to start with a non-georeferenced png (or TIFF) + thumbnail jpg? And / or have separate extractors for 1) converting binary file 2) generating thumbnail and 3) creating georeferenced geoTIFF?

Member

dlebauer commented Apr 21, 2016

@robkooper thanks. That looks like a good first step, additional steps for this issue include:

  1. creating geoTIFF.
    • presumably requires subsequent step with gdal or similar
    • also requires knowledge of image extent
  2. thumbnail generation:
    • Does Clowder already have a thumbnail extractor that we can use?
    • Or, what resolution or max pixel count do you recommend? to handle sensors with different dimensions, we can set max pixel count as convert input.png -resize 4096@ thumbnail.jpg according to imagemagik resize docs.

Would it be worth it to start with a non-georeferenced png (or TIFF) + thumbnail jpg? And / or have separate extractors for 1) converting binary file 2) generating thumbnail and 3) creating georeferenced geoTIFF?

@robkooper

This comment has been minimized.

Show comment
Hide comment
@robkooper

robkooper Apr 21, 2016

Member

yes clowder uses imagemagick to do this. One option which makes it easier is to make an extractor that will look at the metadata for octet-stream files and convert them to an image that is attaches as preview to the file.

Member

robkooper commented Apr 21, 2016

yes clowder uses imagemagick to do this. One option which makes it easier is to make an extractor that will look at the metadata for octet-stream files and convert them to an image that is attaches as preview to the file.

@rachelshekar rachelshekar modified the milestone: May 2016 May 12, 2016

@abby621 abby621 referenced this issue May 20, 2016

Closed

Create full field stitched mosaic #85

1 of 3 tasks complete
@dlebauer

This comment has been minimized.

Show comment
Hide comment
@dlebauer

dlebauer Jun 8, 2016

Member

@TinoDornbusch

@LTBen says 'Tino finished a range of Octave scripts on the control PC such that operators on site can view raw data (Vis, FLIR, PS2, Sensors) and make simple analyses and visualizations.'

Could you please either send these to me or place them in the repository (you can copy and paste them individually here)?

Member

dlebauer commented Jun 8, 2016

@TinoDornbusch

@LTBen says 'Tino finished a range of Octave scripts on the control PC such that operators on site can view raw data (Vis, FLIR, PS2, Sensors) and make simple analyses and visualizations.'

Could you please either send these to me or place them in the repository (you can copy and paste them individually here)?

@TinoDornbusch

This comment has been minimized.

Show comment
Hide comment
@TinoDornbusch

TinoDornbusch Jun 8, 2016

Contributor

@dlebauer OK it is in gantry_data/Lemnatec/OctaveScripts

DO not laugh, just very simple loading procedures knitted with hot needle

Contributor

TinoDornbusch commented Jun 8, 2016

@dlebauer OK it is in gantry_data/Lemnatec/OctaveScripts

DO not laugh, just very simple loading procedures knitted with hot needle

@TinoDornbusch

This comment has been minimized.

Show comment
Hide comment
@TinoDornbusch

TinoDornbusch Jun 8, 2016

Contributor

@dlebauer forgot to mention that Solmaz contributed to by providing the Demosaic algorithm, which is not implemented in Octave

Contributor

TinoDornbusch commented Jun 8, 2016

@dlebauer forgot to mention that Solmaz contributed to by providing the Demosaic algorithm, which is not implemented in Octave

@rachelshekar

This comment has been minimized.

Show comment
Hide comment
@rachelshekar

rachelshekar Jul 7, 2016

Contributor

@abby621 can you please give an update for this issue?

Contributor

rachelshekar commented Jul 7, 2016

@abby621 can you please give an update for this issue?

@pless

This comment has been minimized.

Show comment
Hide comment
@pless

pless Jul 7, 2016

@rachelshekar @max-zilla @dlebauer @abby621 Hi, we have a new person onboarding to this project (Zongyang Li, who will shortly get access to this). While he is getting his clowder development/test environment setup, could we ask that @max-zilla integrate the demosaic script:

https://github.com/terraref/computing-pipeline/blob/master/scripts/stereoImager/demosaic.py

as a clowder extractor, and then provide that as an additional example script? The conference call on the clowder test environment was really good, but having an example to show our new guy how clowder can trigger on the appearance of 2 binary files and a json file in a folder would be helpful.

pless commented Jul 7, 2016

@rachelshekar @max-zilla @dlebauer @abby621 Hi, we have a new person onboarding to this project (Zongyang Li, who will shortly get access to this). While he is getting his clowder development/test environment setup, could we ask that @max-zilla integrate the demosaic script:

https://github.com/terraref/computing-pipeline/blob/master/scripts/stereoImager/demosaic.py

as a clowder extractor, and then provide that as an additional example script? The conference call on the clowder test environment was really good, but having an example to show our new guy how clowder can trigger on the appearance of 2 binary files and a json file in a folder would be helpful.

@max-zilla

This comment has been minimized.

Show comment
Hide comment
@max-zilla

max-zilla Jul 7, 2016

Contributor

@pless @dlebauer @abby621 yes, I can prepare that extractor as an example.

Contributor

max-zilla commented Jul 7, 2016

@pless @dlebauer @abby621 yes, I can prepare that extractor as an example.

@max-zilla

This comment has been minimized.

Show comment
Hide comment
@max-zilla

max-zilla Jul 7, 2016

Contributor

@pless @abby621 I want to test a bit more, but initial version is here:
https://github.com/terraref/computing-pipeline/tree/demosaic_extractor/scripts/stereoImager/extractor

  • the demosaic.py script is unchanged from what is in /scripts, i just put it in extractor folder for simpler importing.
  • listens on files being added to datasets and checks for metadata, _left.bin, _right.bin. the script expects only one set of left/right/metadata per dataset for now
  • when it writes the output JPGs to the dataset and writes COMPLETED as extractor metadata when done, so the extractor won't regenerate the JPGs if more files are added to dataset
  • extractor will attempt to check for metadata in 2 ways: attached to dataset as metadata (eventually we can use this) and for short term, as a metadata.json file in the dataset

Basically, you can run this extractor pointing at rabbitMQ, create a dataset and upload left/right/metadata to that dataset in Clowder pointing at same rabbitMQ, and extractor will add two demosaic'ed JPGs to the dataset.

I might clean this up slightly, but we could have a little phonecall to walk through the code if that would be helpful. fortunately it's fairly simple I think.

Contributor

max-zilla commented Jul 7, 2016

@pless @abby621 I want to test a bit more, but initial version is here:
https://github.com/terraref/computing-pipeline/tree/demosaic_extractor/scripts/stereoImager/extractor

  • the demosaic.py script is unchanged from what is in /scripts, i just put it in extractor folder for simpler importing.
  • listens on files being added to datasets and checks for metadata, _left.bin, _right.bin. the script expects only one set of left/right/metadata per dataset for now
  • when it writes the output JPGs to the dataset and writes COMPLETED as extractor metadata when done, so the extractor won't regenerate the JPGs if more files are added to dataset
  • extractor will attempt to check for metadata in 2 ways: attached to dataset as metadata (eventually we can use this) and for short term, as a metadata.json file in the dataset

Basically, you can run this extractor pointing at rabbitMQ, create a dataset and upload left/right/metadata to that dataset in Clowder pointing at same rabbitMQ, and extractor will add two demosaic'ed JPGs to the dataset.

I might clean this up slightly, but we could have a little phonecall to walk through the code if that would be helpful. fortunately it's fairly simple I think.

@ZongyangLi

This comment has been minimized.

Show comment
Hide comment
@ZongyangLi

ZongyangLi Jul 11, 2016

Contributor

Hi, I am working on fix the GeoTIFF creation, because there is something un-regular with these GeoTIFF creations. I am trying to view images in a google map using tile coordinate. When I finish this, I will start working on a Clowder extrator.

Contributor

ZongyangLi commented Jul 11, 2016

Hi, I am working on fix the GeoTIFF creation, because there is something un-regular with these GeoTIFF creations. I am trying to view images in a google map using tile coordinate. When I finish this, I will start working on a Clowder extrator.

@dlebauer dlebauer removed this from the May 2016 milestone Jul 11, 2016

@dlebauer dlebauer modified the milestones: July 2016, May 2016 Jul 11, 2016

@ZongyangLi

This comment has been minimized.

Show comment
Hide comment
@ZongyangLi

ZongyangLi Jul 14, 2016

Contributor

@max-zilla Hi, I am trying to get my Clowder test environment setup. Here are some problems:

  1. I try to get my machine ip by running "docker-machine ip" in Docker Quickstart Terminal, but I get this error message: "Error: No machine name(s) specified and no "default machine exists". So I simply use "localhost:9000" in my browser, then complete the sign up. Is that Okay for my test environment?
  2. I have run that "wordcount" demo successfully, but when I try "terra.demosaic.py", I got this error message at the time I upload my left/right/metadata:"ERROR: pyclowder.extractors -[5787ec39e4b07f5213e28fc8] error processing Trace back(most recent call last): File "....../pyclowder/extractors.py", line 93, in on_message filename=jbody['filename'] KeyError: 'filename'".

I think I might made a mistake in uploading files to dataset. Could you give me some additional examples to show me how could I provide inputs in Clowder.

Contributor

ZongyangLi commented Jul 14, 2016

@max-zilla Hi, I am trying to get my Clowder test environment setup. Here are some problems:

  1. I try to get my machine ip by running "docker-machine ip" in Docker Quickstart Terminal, but I get this error message: "Error: No machine name(s) specified and no "default machine exists". So I simply use "localhost:9000" in my browser, then complete the sign up. Is that Okay for my test environment?
  2. I have run that "wordcount" demo successfully, but when I try "terra.demosaic.py", I got this error message at the time I upload my left/right/metadata:"ERROR: pyclowder.extractors -[5787ec39e4b07f5213e28fc8] error processing Trace back(most recent call last): File "....../pyclowder/extractors.py", line 93, in on_message filename=jbody['filename'] KeyError: 'filename'".

I think I might made a mistake in uploading files to dataset. Could you give me some additional examples to show me how could I provide inputs in Clowder.

@max-zilla

This comment has been minimized.

Show comment
Hide comment
@max-zilla

max-zilla Jul 14, 2016

Contributor

@ZongyangLi yes, if you were able to access localhost:9000 you have things set up correctly.

That error I believe is due to the version of PyClowder you are using. Currently the pyclowder master branch does not support extractors that operate on datasets (such as the demosaic extractor). Instead you must install my branch:
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/pyclowder/browse?at=refs%2Fheads%2Fbugfix%2FCATS-554-add-pyclowder-support-for-dataset

Easiest way to do this is probably to:

git clone https://opensource.ncsa.illinois.edu/bitbucket/scm/cats/pyclowder.git
cd pyclowder
git checkout bugfix/CATS-554-add-pyclowder-support-for-dataset
python setup.py install

So you pull down PyClowder, switch to my branch that supports datasets, install that version. Then try running demosaic and see if bug is fixed.

Soon my pyclowder branch should become part of main so this won't be necessary, but for now it is. Let me know if that helps!

Contributor

max-zilla commented Jul 14, 2016

@ZongyangLi yes, if you were able to access localhost:9000 you have things set up correctly.

That error I believe is due to the version of PyClowder you are using. Currently the pyclowder master branch does not support extractors that operate on datasets (such as the demosaic extractor). Instead you must install my branch:
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/pyclowder/browse?at=refs%2Fheads%2Fbugfix%2FCATS-554-add-pyclowder-support-for-dataset

Easiest way to do this is probably to:

git clone https://opensource.ncsa.illinois.edu/bitbucket/scm/cats/pyclowder.git
cd pyclowder
git checkout bugfix/CATS-554-add-pyclowder-support-for-dataset
python setup.py install

So you pull down PyClowder, switch to my branch that supports datasets, install that version. Then try running demosaic and see if bug is fixed.

Soon my pyclowder branch should become part of main so this won't be necessary, but for now it is. Let me know if that helps!

@ZongyangLi

This comment has been minimized.

Show comment
Hide comment
@ZongyangLi

ZongyangLi Jul 14, 2016

Contributor

@max-zilla Thank you! That's really helpful. Clowder starts to process when left/right/metadata are ready. But it choose an incorrect file as an input dataset. I got these output information in the terminal:
"img_left: /var/folders/rw/4bn7m8x90c1_vjxlw94gxk6h0000gn/T/tmpHDehoV/1_left.bin_57880e72e4b07f52668dee40/_metadata.json

So how could I get the file path from "parameters" in "process_dataset" function?

Contributor

ZongyangLi commented Jul 14, 2016

@max-zilla Thank you! That's really helpful. Clowder starts to process when left/right/metadata are ready. But it choose an incorrect file as an input dataset. I got these output information in the terminal:
"img_left: /var/folders/rw/4bn7m8x90c1_vjxlw94gxk6h0000gn/T/tmpHDehoV/1_left.bin_57880e72e4b07f52668dee40/_metadata.json

So how could I get the file path from "parameters" in "process_dataset" function?

@max-zilla max-zilla referenced this issue Jul 15, 2016

Closed

Add hyperspectral data extractor to pipeline #81

0 of 4 tasks complete
@ZongyangLi

This comment has been minimized.

Show comment
Hide comment
@ZongyangLi

ZongyangLi Jul 20, 2016

Contributor

@max-zilla I am still working on this problem. Could you please give me more examples that how to access all datasets since I can not run the demosaic extractor successfully.

Contributor

ZongyangLi commented Jul 20, 2016

@max-zilla I am still working on this problem. Could you please give me more examples that how to access all datasets since I can not run the demosaic extractor successfully.

@ZongyangLi

This comment has been minimized.

Show comment
Hide comment
Contributor

ZongyangLi commented Jul 22, 2016

@max-zilla

This comment has been minimized.

Show comment
Hide comment
@max-zilla

max-zilla Aug 4, 2016

Contributor

This discussion has moved to #133 and the code is written. closing.

Contributor

max-zilla commented Aug 4, 2016

This discussion has moved to #133 and the code is written. closing.

@max-zilla

This comment has been minimized.

Show comment
Hide comment
@max-zilla

max-zilla Sep 6, 2016

Contributor

This issue was moved to terraref/extractors-stereo-rgb#1

Contributor

max-zilla commented Sep 6, 2016

This issue was moved to terraref/extractors-stereo-rgb#1

@rachelshekar rachelshekar added sensor-data and removed sensor data labels Jan 3, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment