When downloading an order, manifest.json sometime overwrites contents of another file

[jankolmas]

Sometimes, one of the files in the order is overwritten with the contents of the manifest.json, while keeping its filename. I did some digging, and I think I understand why.

In the file porder/downloader.py, line 167, when the program encounters the file manifest.json, it fetches it, reads it, and for each file contained in the manifest, and attempts to create a local_path (it is not clear to me why, but that is fine). The problem is that when the code continues on line 184, the local_path variable is actually not the path of the manifest file, it is the path of the last file contained in the manifest (for example the udm2 file). [let's call it FILE] The program then attempts to download manifest.json, but save it as FILE.

Within a given order, the files (for example: analytic tiff, udm, udm2, metadata xml, metadata json and manifest json) seem to be in a random sequence.

1. If manifest.json is after FILE in the sequence, it is fine, the program will print "File already exists SKIPPING" and doesn't download the manifest.

2. If manifest.json is before FILE in the sequence, it downloads manifest.json, and saves it as FILE. When it then tries to download the real FILE, it is skipped with the message "File already exists SKIPPING"

I think that the fix is quite simple, remove the else statement on line 182, remove one indent from line 183. I could be wrong though, I did not test it.

Cheers,
Jan

[jankolmas]

Hi, just a note that I tested the proposed fix, and it seems to work fine. My team would really appreciate it if you could have a look at this soon, since we use porder in a production setting, and we currently have to use a workaround. Thanks a lot, we appreciate the work you put into this package!

[samapriya]

So here's a couple of things that are happening the downloader never downloads or attempts to download the manifest.json file, it simply uses it to get to the file names hence in line 184 it only passes out those files that don't end with manifest.json. So I am unclear here, it loses this functionality if the else is deleted in line 183 and ident is adjusted so it will try to download a manifest.json file. Can you send me a file where it tries to download the manifest.json file and saves it with a different file name?

[jankolmas]

Hi Samapriya,
The command I run is porder download --url "https://api.planet.com/compute/ops/orders/v2/116f0a4a-2887-447e-a194-23ed14c69be4" --local "/serverdata/svs/tst"

These are the files I get:

And this is the metadata file (I had to rename it from .json to .txt to conform to GitHub upload rules)
20210524_095547_1038_metadata.txt

[jankolmas]

Let me show you my reasoning.

Line 153: Loop over the files listed in the response to the GET request. (these files include manifest.json). Let's call this MAIN LOOP
Line 167: Let's take the case when manifest.json file is encountered, ...
Line 173: ...loop over the files listed in the manifest (these files do not include manifest.json), and create the local_path variable. Let's call this SUB LOOP Note that this variable is overwritten with every cycle of the SUB LOOP, so we when the SUB LOOP is finished, local_path has the value of the last item in the manifest.
Line 186: We are still in the MAIN LOOP and the "current" file is manifest.json. Therefore, the variable "url" contains the link to the manifest.json file
Line 190: We download the manifest.json file, but save it under a different filename, because the local_path variable was set to the filename of a different file on line 175.

Does that make sense?

[samapriya]

Thanks for the detailed comment, it would also be great to have you star the repo and cite the work as you are using it since you said you are using this for production code. This helps build stats, increases visibility, and encourages continuous development, would love to know which company is using this code is it SuperVision or something else? I will get the fix in soon

[jankolmas]

Will do! Indeed, we are using porder at SuperVision Earth. Thanks a lot for maintaining the repo, it saves us a lot of work :)

[samapriya]

Now updated to porder v0.8.4 issue has been resolved based on discussion,thanks for feedback