Head metadata .xml makes reference to downloaded tile only.

[wouellette]

Hi,

I will post my question as a new issue, although I had initially replied to a closed post from Safalabolo:

Safalabolo:
"Unfortunately, the processing by SEN2COR of granule downloaded fails, contrary to what happens for the same granule selected from the entire acquisition (zip file) downloaded via a simple download manager."

Your answer: "I will modify the downloader to produce the same structure, with dummy files, so that Sen2Cor will process them."

The head .xml metadata file still doesn't match the same structure as it does would you download it from the esa hub directly. It's fine if you download only one tile and do the sen2cor and subsequent processing on that one and single tile, but if you download one tile, and then some time later another one of that same image, the .SAFE folder will be completed by the new granule(s), but the head metadata .xml file will be replaced by a metadata file referring to that new tile only. What you refer to as the "dummies" should be there to mimic the structure of the scihub data.

I am trying to see what needs be done in your code, but I am unfortunately not too handy in java.

Thanks for the effort in all cases and hope to hear back soon.

[kraftek]

It would be enough to put the two tile ids as argument of --tiles parameter. The already downloaded one will not be downloaded again, the second one will be downloaded and the product metadata will reflect both.

[wouellette]

As you mentioned, the new granules are downloaded and the already downloaded one remains as it is, but the general metadata .xml is not updated with the newly downloaded tiles and still only refers to the single tile which I had previously downloaded, meaning that only that tile shows when opened in SNAP or any other software.

For the S-2 images which are newly downloaded (i.e. no previous tiles of that same image had been downloaded before), the general metadata .xml is created conform to the number of tiles referred to in the argument --tiles, which is great. However, the "updating" seems to be the issue if part of the image was previously downloaded in another instance. I have hundreds of images, and looking through all of them to see which .xml files are incorrect/corrupted would take forever.

Looking for a smooth solution, but if there is no such thing I will do it the hard way, can't be that bad :)

Thanks in advance

[kraftek]

What if you supply all the tileIDs you need for a product for which you have previously downloaded just a part of its tiles?
For example, if you have previously download product X with only the tile 35JMK, next time you try with the same product X with the tiles 35JMK and 35JML.
Normally, with the latest version from Git, the product metadata should be overwritten each time.

[wouellette]

I got the latest version from git, but now get the following error after packaging and assembling as described:

Exception in thread "main" java.lang.NullPointerException
at java.util.Properties$LineReader.readLine(Properties.java:434)
at java.util.Properties.load0(Properties.java:353)
at java.util.Properties.load(Properties.java:341)
at ro.cs.products.ProductDownloader.(ProductDownloader.java:61)
at ro.cs.products.sentinel2.SentinelProductDownloader.(SentinelProductDownloader.java:87)
at ro.cs.products.Main.main(Main.java:428)

The command line I use is the following:
java -jar /home/s_admin/awsdownload/target/ProductDownload-bin/ProductDownload/ProductDownload.jar --sensor S2 -o /nas --relative.orbit 51 --tiles 31UDT 31UDS 31UET 31UES 31UER 31UFT 31UFS 31UFR 31UFQ 31UGS 31UGR -l 1000 --startdate 2015-01-01 --cloudpercentage 80 --store AWS --user ****** --password *****

[wouellette]

I have a feeling it is not accepting a linux syntax for the output folder: "--out /nas"

[wouellette]

I've traced the error up to ProductDownloader.java, where it seems to have issues reading the "download.properties" file. But it's weird because if the resource read was indeed Null, then an error should be thrown and printed as indicated by the "Catch (IOException e)" line, but instead i get this NullPointerException in "Main".

Just trying to give some hints to where the problem could come from based on what I could find online, but still can't seem to find a solution.

[wouellette]

The last commit solved the issue.
Cheers

[kraftek]

Glad to hear J

Cheers.

From: wouellette [mailto:notifications@github.com]
Sent: Friday, November 18, 2016 4:07 PM
To: kraftek/awsdownload awsdownload@noreply.github.com
Cc: Cosmin Cara kraftek@c-s.ro; Comment comment@noreply.github.com
Subject: Re: [kraftek/awsdownload] Head metadata .xml makes reference to downloaded tile only. (#5)

The last commit solved the issue.
Cheers

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub #5 (comment) , or mute the thread https://github.com/notifications/unsubscribe-auth/AHQBsNfEfgSJRMacGuv4YQRpqWKWnA0cks5q_bDsgaJpZM4KLEt_ . https://github.com/notifications/beacon/AHQBsMl-Z1JIJN0xCZEzvhn4BdQPwIxeks5q_bDsgaJpZM4KLEt_.gif