subdividing FL for wv2 processing project #3
Comments
|
Hi Tylar,
Sorry for the delay. I'm back from D.C. last week, but working from home
today on the NERRS and NSF grant details. I agree with the FCMaP divisions
and will try to recreate those files from which to select images. I'm
thinking of creating directories for each division, and subdirectories for
each year within each division. Where should I place the images? Or should
I go about the file collection a different way?
Best,
Matt
…On Mon, Jun 4, 2018 at 9:45 AM, Tylar ***@***.***> wrote:
@mjm8 <https://github.com/mjm8> : I think we spoke briefly about dividing
up the area we intend to cover under the microsoft grant. You may already
have something better figured out, but I was thinking we could use the same
subdivisions as FCMaP
<https://www.usgs.gov/media/images/graphic-location-map-shows-six-regions-fcmap>
to divide up FL.
Based on those subdivisions I am thinking we run "big bend" and "West FL
Pen." on our servers & do the others in microsoft's cloud.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#3>, or mute the thread
<https://github.com/notifications/unsubscribe-auth/Af6OKMx9K9-gokZfkOBMmELXLzdvGHLiks5t5TnmgaJpZM4UZHbN>
.
--
Matt McCarthy, Ph.D.
Biological Oceanography
College of Marine Science
University of South Florida
140 7th Avenue South
St Petersburg, FL 33701-5016
727-553-1186
|
|
It is relatively trivial for me to re-organize images once I have indexed them, so we can use whatever organization paradigm works best for you (and others(?) accessing wv2 data). We can even keep using That being said, I would most prefer to organize the images using this imars-objects spec - which is very near to what you propose. In this case I think it would look something like: it might also be nice to split up the different files by subdividing wv2 somehow. eg: ...but only if that subdivision won't cause any issues for your (or others) processing routines. |
|
That sounds good. I'll start working on it this week.
…On Mon, Jun 11, 2018 at 11:39 AM, Tylar ***@***.***> wrote:
It is relatively trivial for me to re-organize images once I have indexed
them, so we can use whatever organization paradigm works best for you (and
others(?) accessing wv2 data). We can even keep using /extra_data if
desired.
That being said, I would most prefer to organize the images using this
imars-objects spec
<https://github.com/USF-IMARS/IMaRS-docs/tree/master/docs/management_data/imars-objects>
- which is very near to what you propose. In this case I think it would
look something like:
/big_bend
/wv2
/...
/2017
/2018
/fl_se
/wv2
/...
/2017
/2018
/fl_ne
/wv2
/...
/2017
/2018
/monroe
/wv2
/...
/2017
/2018
/panhandle
/wv2
/...
/2017
/2018
/west_fl_pen
/wv2
/...
/2017
/2018
it might also be nice to split up the different files by subdividing wv2
somehow. eg:
/big_bend
/wv2_m1bs
/wv2_p1bs
...but only if that subdivision won't cause any issues for your (or
others) processing routines.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#3 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/Af6OKByWg_C82zoBOZFJkB_kZTjSGx93ks5t7o8cgaJpZM4UZHbN>
.
--
Matt McCarthy, Ph.D.
Biological Oceanography
College of Marine Science
University of South Florida
140 7th Avenue South
St Petersburg, FL 33701-5016
727-553-1186
|
|
The initial tests have gone well enough. Still a few bugs to iron out, but nothing major. I would like to start processing the big bend region on our servers here. I think the easiest way to do this will be to set up ftp downloads from digital globe as we were before, using the filename convention What do you think @mjm8 ? |
enables moving forward with plans in USF-IMARS/wv-land-cover#3
|
I'm glad you're ready to start processing, though we should also take
advantage of the images we already have, which includes all of the Big Bend
for 2014/2015. Can we start with those to test, and move to updating our
database when that goes smoothly?
The other thing we need to consider for Azure and NSF is that our old PGC
contacts just informed me that they are no longer filling our requests for
large batches of imagery. They said they're working on an alternative for
us, but we may need to either find our own alternative for large scale
image requests, or start downloading them individually through the DG
portal. Maybe we need an intern? Enrique happens to be here right now and
said he has an intern for other labs stuff who wants to get involved in
remote sensing. I'll send a separate email to you and him now on that.
In the meantime, let me know if we can use the existing imagery, and where
I should put that as well as how many images (2? 10? 100?)
…On Wed, Jun 27, 2018 at 5:20 PM, Tylar ***@***.***> wrote:
The initial tests have gone well enough. Still a few bugs to iron out, but
nothing major.
I would like to start processing the big bend region on our servers here.
I think the easiest way to do this will be to set up ftp downloads from
digital globe as we were before, using the filename convention
wv2_YYYY-MM-DDThhmm_big_bend.zip, where the datetime used is the time the
order is placed.
What do you think @mjm8 <https://github.com/mjm8> ?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#3 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/Af6OKM29kDxXdNJiHW6ZulkDPcmOdabUks5uA_ctgaJpZM4UZHbN>
.
--
Matt McCarthy, Ph.D.
Biological Oceanography
College of Marine Science
University of South Florida
140 7th Avenue South
St Petersburg, FL 33701-5016
727-553-1186
|
I think we need a few 100 to keep the servers busy while I explore optimizations.
Ouch. If this is the case I can get creative with automating it, but it would be much better for everyone if DG exposed an API. We'll talk more on that later I suppose.
Oh, perfect! Ultimately we need to "load" them using this imars-etl tool I have been working on. This puts the metadata into a database and copies the file to |
|
It looks like I've got 454 NTF and XML files each in one directory in
Tampa. I can scp those wherever you like.
…On Thu, Jun 28, 2018 at 10:47 AM, Tylar ***@***.***> wrote:
how many images (2? 10? 100?)
I think we need a few 100 to keep the servers busy while I explore
optimizations.
start downloading them individually through the DG portal
Ouch. If this is the case I can get creative with automating it, but it
would be much better for everyone if DG exposed an API. We'll talk more on
that later I suppose.
the images we already have, which includes all of the Big Bend for
2014/2015 [ ... ] and where I should put that
Oh, perfect! Ultimately we need to "load" them using this imars-etl
<https://github.com/USF-IMARS/imars-etl> tool I have been working on.
This puts the metadata into a database and copies the file to
/srv/imars-objects/big_bend/wv2/ so all the servers can find them. Do you
have a list of the files or are the all in a directory?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#3 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/Af6OKIvO4OQKNc1wzScO8ecWGWz2gDHmks5uBOx0gaJpZM4UZHbN>
.
--
Matt McCarthy, Ph.D.
Biological Oceanography
College of Marine Science
University of South Florida
140 7th Avenue South
St Petersburg, FL 33701-5016
727-553-1186
|
|
This should work, but you might want to test it on a small number of files first. The ideal location would be |
|
My password for that directory is not working. Should it be the same as my
seashell password? Or do I need to set one up?
…On Thu, Jun 28, 2018 at 11:52 AM, Tylar ***@***.***> wrote:
This *should* work, but you might want to test it on a small number of
files first. The ideal location would be ***@***.***
edu:/thing2/imars-objects/big_bend/wv2/2014 and .../2015
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#3 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/Af6OKJCWZl_HKyxdznONz38EYLB_BDHkks5uBPvZgaJpZM4UZHbN>
.
--
Matt McCarthy, Ph.D.
Biological Oceanography
College of Marine Science
University of South Florida
140 7th Avenue South
St Petersburg, FL 33701-5016
727-553-1186
|
|
Should be the same. can you ssh to |
|
Nope. Password doesn't work.
…On Thu, Jun 28, 2018 at 12:03 PM, Tylar ***@***.***> wrote:
Should be the same. can you ssh to thing2.marine.usf.edu?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#3 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/Af6OKChW8rMilkHFLg2rhpdnuJvAunuqks5uBP5GgaJpZM4UZHbN>
.
--
Matt McCarthy, Ph.D.
Biological Oceanography
College of Marine Science
University of South Florida
140 7th Avenue South
St Petersburg, FL 33701-5016
727-553-1186
|
|
Ah, okay. I guess it is not working. Noted. Use It will end up the in the same place, just a bit more network usage. |
|
Transferring 2014...
2015 next.
…On Thu, Jun 28, 2018 at 12:43 PM, Tylar ***@***.***> wrote:
Ah, okay. I guess it is not working. Noted
<https://github.com/USF-IMARS/server-status/issues/104>.
Use ***@***.***:/srv/imars-objects/big_bend/wv2/2014
then.
It will end up the in the same place, just a bit more network usage.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#3 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/Af6OKEI3pceV9-TU-uHl3cQK2GL6wGRAks5uBQfMgaJpZM4UZHbN>
.
--
Matt McCarthy, Ph.D.
Biological Oceanography
College of Marine Science
University of South Florida
140 7th Avenue South
St Petersburg, FL 33701-5016
727-553-1186
|
|
How go the transfers? |
|
I thought I transferred all the 2014 and 2015 images of Big Bend. I'll
organize and send the rest of what we have when I'm back next week, but for
new images we either need the intern or Frank's connections with NASA.
How is the processing going?
…On Tue, Jul 17, 2018, 8:30 AM Tylar ***@***.***> wrote:
How go the transfers?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#3 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/Af6OKGdEWXYBEiH8KlgFLn6eMdcVNK_wks5uHgL-gaJpZM4UZHbN>
.
|
|
I just fixed an issue that was killing 2/3 of the jobs. You can check on the running jobs in airflow here, and the output files are being saved to |
enables moving forward with plans in USF-IMARS/wv-land-cover#3
@mjm8 : I think we spoke briefly about dividing up the area we intend to cover under the microsoft grant. You may already have something better figured out, but I was thinking we could use the same subdivisions as FCMaP to divide up FL.
Based on those subdivisions I am thinking we run "big bend" and "West FL Pen." on our servers & do the others in microsoft's cloud.
The text was updated successfully, but these errors were encountered: