ISSUE-148 Add option to burn features from multiple vectors onto raster

[theoilie]

Closes #148 but is still a WIP. It seems to be running without errors on my subset image, but I haven't yet found a reasonably-sized hyperspectral image with close proximity to dams and reservoirs. I'm still looking around on https://aviris.jpl.nasa.gov/alt_locator/, but it takes time to download the large files and then classify them. So far I've been manually importing the files into QGIS and then adding the dams and reservoirs vector layers to see if there's anything nearby. I'll let you know when I find a good example that can demonstrate that streams/lakes/rivers are working with dams/reservoirs on the same image.

[coveralls]

Coverage increased (+0.5%) to 63.211% when pulling 07ccb18 on ISSUE-148 into c685b4d on master.

[lewismc]

Nice work... please keep us updated on the trial and error approach with the alt_locator. Wouldn't it just be great if we were able to search via proximity to some geographic feature?

[theoilie]

I tried to come up with something more efficient than trial and error: calculating the latitude and longitude of the dams/reservoirs using the Field Calculator in QGIS. However, I found that the bounding box on the alt_locator had some issues so this approach is still pretty inaccurate. I have to cross-reference the general location with Google Maps and then find a nearby image in the alt_locator (and even then, the landmarks shown in the alt_locator don't line up with Google Maps so it's difficult to find corresponding locations and gauge their proximity to dams and reservoirs).

Long story short - I've been playing around with a couple of images that I think have mining activity and are close to dams and reservoirs. Still working on confirming this, though.

Regarding searching by feature @lewismc, that would be amazing. I haven't forgotten about the clustering approach you mentioned last week with centroids and searching more efficiently than pixel-by-pixel. I'll bring it up in the tagup tomorrow so I can flesh it out enough to create an issue.

[lewismc]

Got it, thanks

[lewismc]

Hi @Lactem I think I ended up figuring out a better workflow.

• In QGIS I installed the OSM Place Name and OpenLayers plugins
• I then loaded the GRaND reservoirs shapefile and the wikimedia labelled layer (from the OpenLayers plugin)

This enabled me to look at flight lines in the AVIRISNG alt_locator which capture data over significant water bodies and then go over to QGIS and plug in the location name to see if there was any overlap with GRaND.

I ended up getting the following results. Please start with the latter (ang20190623t220015). It is much smaller than the first one.

[lewismc]

Hi @Lactem if you also use #191 you will be able to speed up product generation. Thanks

[theoilie]

@lewismc Thanks for this. I didn't see it until just now and I've already tried another 2 files (1 was in close proximity but didn't have mining activity after I classified it). I'll try these files out today.

[lewismc]

ack

[theoilie]

@lewismc The download link is not present from the map. I ended up finding it from the database view, but it's currently broken. I browsed around the FTP server and noticed that there's simply no data uploaded from 2019 (perhaps the plan is to release it at the end of the year?). Anyway, I found the 2017 data while browsing, so I'm currently classifying the larger image you mentioned with #191. So far I'm surprised because the RGB output is in black and white only, which makes me worry that the final classified product will be incorrect -- but I'll let it finish running and see what happens.

This is not directly related, but I noticed that Google Fusion Tables is being taken down in less than a month (link). Just thought I'd throw that out there since it looks like the alt_locators depend on it.

[lewismc]

I browsed around the FTP server and noticed that there's simply no data uploaded from 2019 (perhaps the plan is to release it at the end of the year?)

I emailed my colleagues Sarah Lundeen and Winston Olson-Duvall at JPL with the request for the data... we should have it soon.

Anyway, I found the 2017 data while browsing, so I'm currently classifying the larger image you mentioned with #191.

Great

So far I'm surprised because the RGB output is in black and white only, which makes me worry that the final classified product will be incorrect -- but I'll let it finish running and see what happens.

We'll the RGB available from the alt_locator is also greyscale so I wonder if the scene was messed up... there's only one way of finding out.
One other thing I discovered is that sometimes QGIS loaded ratsers in greyscale (maybe something to do with memory usage or UI performance). It was however possible to 'correct' this by viewing the layer properties and adjusting some settings. @aheermann covered this somewhere but I can't remember where.

This is not directly related, but I noticed that Google Fusion Tables is being taken down in less than a month (link). Just thought I'd throw that out there since it looks like the alt_locators depend on it.

I've never even used fusion tables, I knew nothing about it. I'll let the AVIRIS team know when they respond with the scenes. Thanks

[theoilie]

This was the mineral classified image. It's very close to multiple dams and reservoirs, but unfortunately the mining classified image is empty (there's no mining activity). Maybe we'll have better luck with the next image.

[lewismc]

@Lactem

Host: avng.jpl.nasa.gov
Port: 21
Paths: /pub/LMcgibbney
User: anonymous (or leave blank)

We recommend connecting with a client like Filezilla or Cyberduck. You can also see these files in many browsers (like Chrome) using the linksftp://avng.jpl.nasa.gov/pub/LMcgibbney

[lewismc]

...This was the mineral classified image. ...

Thanks so much for running this scene, the results are unfortunate. Please try with the new scene I posted above.

[theoilie]

@lewismc I'll download and classify this tonight, thanks.

[theoilie]

This screenshot of the latest scene shows that there were plenty of minerals, but unfortunately no mining activity again. I can look around more for another scene using the OpenLayers plugins you mentioned.

[lewismc]

Here's a thought... although this is bad for providing a use case, it is in reality a REALLY good thing.!
Thanks for continuing the pursuit.

[theoilie]

@lewismc Haha that's a good point, it's good that it's so hard to find mining activity near reservoirs. I'll report any new findings as they arise.

[lewismc]

Any luck with those new scenes ?

[theoilie]

No luck just yet. While this is not specifically for coal mining, from my understanding the tailings from potash products are just as toxic to the environment. This whole paper is extremely long, and section 3.3.3.1 is what strikes me as the most useful to us:

Solid potash tailings consist of sodium chloride with a few per cent of other salts and insoluble
materials such as clay and anhydrite (see figure ‘sylvinite tailings’). Hard salt tailings
additionally contain about 5 % kieserite (see figure ‘hard salt tailings’).

Also section 3.4.3.1:

Typically tailings from the Ruhr, Saar and Ibbenbüren areas in Germany consist of 55 - 60 %
clay shale, 30 - 40 % sandy clay shale and 5 - 15 % sandstone (Prosper-Haniel mine) [79, DSK,
2002].

Also section 3.2.6.1:

Limestone tailings are a mixture of calcite, dolomite, wollastonite and other very insoluble
silicates and very small amounts of heavy metals.

From my research, halite makes up 89%-94% of tailings. Also, although kieserite makes up a low percentage of tailings (and only for potash as far as I know), it's the second-most common mineral. I've read elsewhere that muscovite is often observed in high levels in tailings.

Also, one major caveat is that this research studied mines in European countries, whereas we're classifying data in the U.S. and Canada and maybe Mexico. So I'm not sure how the minerals differ by region.

Another problem is that these are not spectroscopy analyses so the only mineral I can add to the PROXY_CLASS_NAMES for mining correlation is "Kieserite KIEDE1.a crse gr NIC4cc AREF."

Final issue -- I realized that the mineral example classifies minerals with spectral library v7, but the mining example uses v6. This means that there could have actually been mining activity in some of the previous scenes, but they would've been missed in classification because the names of mining-indicative materials vary by version. For future tests I'll use v7 across the board.

Another potentially useful source -- I encountered a number of research papers that claimed to have analyzed the spectral characteristics of coal mining or other mining. However, I don't have access to all of these. @lewismc if you have access to Taylor & Francis, this one paper in particular looks promising. Finally, this publicly available source from a USC student is a very well written case study that overlaps with Pycoal a lot. The paper says they used 3 sulfates to identify mining activity: Jarosite (KFe3+3(OH)6(SO4)2), Alunite (KAl3(SO4)2(OH)6), and Muscovite (KAl2(AlSi3O10)(F,OH)2). I should be able to search for these in the Spectral library and find their exact names.

[lewismc]

The following is very relevant

Tailings and waste-rock consist of the different natural minerals found in the mined rock. In the unmined rock, often situated deep below the ground level, the reactive minerals are protected from oxidation. In oxygen-free environments, such as in deep groundwater, the sulphide minerals are thermodynamically stable and have low chemical solubility. Deep groundwater in mineralised areas, therefore, often has a low metal content. However, when excavated and brought to the surface, the exposure to atmospheric oxygen starts a series of bio-geo-chemical processes that can lead to production of acid mine drainage. Hence, it is not the content of metal sulphides in itself that is the main concern, but the combined effects of the metal sulphide content and the exposure to atmospheric oxygen. The effect of exposure increases with decreasing grain size and, therefore, increased surface area. Hence the sulphides in the finely ground tailings are more prone to oxidation [14, Höglund, 2001].

Tailings and waste-rock are normally composed of a number of minerals, of which the sulphides only constitute one part, if present at all. Therefore, if sulphide oxidation occurs in mining waste, the acid produced may be consumed by acid consuming reactions in varying degrees, depending on the acid consuming minerals available. If carbonates are present in the mining waste, pH is normally maintained as neutral, the dissolved metals precipitate and thus are not transported to the surrounding environment to any significant degree. Other acid consuming minerals include alumino-silicates. The dissolution of alumino-silicates is kinetically controlled and cannot normally maintain a neutral pH in the drainage.

The interaction between the acid producing sulphide oxidation and the acid consuming dissolution of buffering minerals determines the pH in the pore water and drainage, which in turn influences the mobility of metals. If the readily available buffering minerals are consumed, the pH may drop and ARD will then occur.

The release of ARD to surface- and groundwater deteriorates the water quality and may cause a number of impacts, such as depletion of alkalinity, acidification, bioaccumulation of metals, accumulation of metals in sediments, effects on habitats, elimination of sensitive species and unstable ecosystems. The chemical processes of acid generation and acid consumption are explained in Section 2.7

[lewismc]

Links I'll be using to augment the literature review
https://en.wikipedia.org/wiki/Environmental_impact_of_mining
https://www.usgs.gov/faqs/how-does-mine-drainage-occur?qt-news_science_products=0#qt-news_science_products
https://wvmdtaskforce.files.wordpress.com/2016/04/2016-etd30-pfz-a.pdf
https://www.sciencedirect.com/topics/chemistry/acid-mine-drainage
https://cfpub.epa.gov/si/si_public_record_Report.cfm?Lab=NRMRL&dirEntryID=174426
https://www.thoughtco.com/what-is-acid-mine-drainage-1204134
http://www.sosbluewaters.org/epa-what-is-acid-mine-drainage%5B1%5D.pdf
https://www.sciencedirect.com/topics/earth-and-planetary-sciences/acid-mine-drainage
https://en.wikipedia.org/wiki/Acid_mine_drainage
http://groundtruthtrekking.org/Issues/MetalsMining/AcidMineDrainage.html

From what I found, it brings the additional list to the following

Mineral	Present in USGS Base Spectra (splib07a)	link
muscovite	Yes	https://crustal.usgs.gov/speclab/QueryAll07a.php?quick_filter=muscovite#
sylvinite	No
halite	Yes	https://crustal.usgs.gov/speclab/QueryAll07a.php?quick_filter=halite#
kieserite	Yes	https://crustal.usgs.gov/speclab/QueryAll07a.php?quick_filter=kieserite#
calcite	Yes	https://crustal.usgs.gov/speclab/QueryAll07a.php?quick_filter=calcite#
dolomite	Yes	https://crustal.usgs.gov/speclab/QueryAll07a.php?quick_filter=dolomite#
wollastonite	Yes	https://crustal.usgs.gov/speclab/QueryAll07a.php?quick_filter=wollastonite#
sulphuric acid	No
sulphide	No
arsenic	No
mercury	No
lead	Yes	https://crustal.usgs.gov/speclab/QueryAll07a.php?quick_filter=lead#
cadmium	Yes	https://crustal.usgs.gov/speclab/QueryAll07a.php?quick_filter=cadmium#
pyrite (iron sulfide)	Yes	https://crustal.usgs.gov/speclab/QueryAll07a.php?quick_filter=pyrite#
gypsum	Yes	https://crustal.usgs.gov/speclab/QueryAll07a.php?quick_filter=gypsum#

[lewismc]

Hi @Lactem please rebase with master to resolve conflicts.

[theoilie]

@lewismc Please note that the failing tests seem to have been introduced from the master branch. I also updated the Spectral library version to 7 and will test the new proxy classes momentarily.

[lewismc]

Thanks @Lactem I think this is nearly good to merge.

@lewismc Please note that the failing tests seem to have been introduced from the master branch

That's fine @mjn0898 is fixing that in #201

I also updated the Spectral library version to 7 and will test the new proxy classes momentarily.

Excellent. Did you ever manage to run over the Fort McMurray flight lines?

Also, can you please add the GRanD shapefiles to the test directory. I can't see them included in the pull request. Thanks

[theoilie]

Also, can you please add the GRanD shapefiles to the test directory. I can't see them included in the pull request. Thanks

Just added them but then realized the reason I didn't add them earlier: the reservoirs shapefile is over 50MB. Would you prefer I keep them or remove them and add a download link in the examples README? If we do it the latter way then nosetests won't be able to verify proximity functionality.

Did you ever manage to run over the Fort McMurray flight lines?

I've been working with ang20170811t212940, which is part of that case study, but I haven't had a chance to run the other scenes yet. I'm still testing out the proxy classes and figuring out an issue with their spacing and encoding.

[lewismc]

Would you prefer I keep them

Yes. Please use the following functionality to exclude examples and tests directories from being included in the package distribution

https://stackoverflow.com/questions/26545668/setup-py-sdist-exclude-packages-in-subdirectory

I've been working with ang20170811t212940, which is part of that case study, but I haven't had a chance to run the other scenes yet.

OK great thanks

I'm still testing out the proxy classes and figuring out an issue with their spacing and encoding.

Got it. Yes these are a bit finicky but I appreciate you adding more... this will add significantly to detection capability.

[theoilie]

exclude examples and tests directories from being included in the package distribution

Will do. After more manual inspection I'm pretty sure there's no mining activity in the scene I've been using. I added a lot of proxy classes, so I'll try classifying a different scene from the Fort McMurray campaign.

[lewismc]

@Lactem please push a trivial update to this branch so that TravisCI is rerun. I would like to see all tests pass before we merge into master. Thanks

[theoilie]

Update: I didn't see any of the proxy minerals present in ang20170811t212940 or ang20170811t202652. I even tried manually going into QGIS and removing all minerals that weren't proxies from the mineral-classified image, in order to double-check that the mineral image really was blank after trimming it down to only the proxies (it was blank).

I looked at the JPEG preview of ang20170811t210417 and see something that looks like heavy mining activity (I think), so I'm currently downloading and classifying this third scene to see if I can verify the proximity functionality.

please push a trivial update to this branch so that TravisCI is rerun

I'll push very soon.