Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create biome lookup #23

Closed
8 tasks done
oubiwann opened this issue Sep 18, 2019 · 28 comments · Fixed by #27
Closed
8 tasks done

Create biome lookup #23

oubiwann opened this issue Sep 18, 2019 · 28 comments · Fixed by #27

Comments

@oubiwann
Copy link
Member

oubiwann commented Sep 18, 2019

  • Gather general biome data
  • Add temperature, elevation, and precipitation scales
  • Add temperature, elevation, and precipitation maps
  • Add a temperature map adjusted for elevation
  • Create a biome-mapping based on one of the accepted models in the field
  • Create one or more data structures to implment this mapping in Clojure
  • Generate a new world map with all the biomes
  • Analyze results to see if global temps/precip maps need to be adjusted
@oubiwann
Copy link
Member Author

oubiwann commented Sep 18, 2019

2000px-Altitudinal_zones_of_Alps_mountains-extended_diagram svg

2000px-Lifezones_Pengo

Vegetation

Climate_influence_on_terrestrial_biome

world-biome-map-columbia-nasa-wwf

Cont_w_leg-med

Köppen-Geiger_Climate_Classification_Map_medium

@oubiwann
Copy link
Member Author

Elevation scale added with commit d772e3c (other related work updated in their respective tickets, e.g., #17 & #22)

@oubiwann
Copy link
Member Author

oubiwann commented Sep 22, 2019

Improved scales for elevation and precipitation using indexed map colors and exported palettes:
Screen Shot 2019-10-07 at 1 03 47 AM

Screen Shot 2019-10-07 at 12 54 05 AM

Screen Shot 2019-10-06 at 12 54 24 PM

@oubiwann
Copy link
Member Author

Made some good progress tonight; should have an elevation-adjusted temperature map soon ...

@oubiwann
Copy link
Member Author

oubiwann commented Sep 28, 2019

It's not the prettiest, but it's more than nothing (high altitudes now have lower temperatures):
001-mercator-offset-elevation-temperature

@oubiwann
Copy link
Member Author

oubiwann commented Oct 6, 2019

New map for temps:
001-mercator-offset-temperature-5

@oubiwann
Copy link
Member Author

oubiwann commented Oct 7, 2019

Some precipitation+temperature-based approaches for biome/ecoregion/climate zone calculations:

  1. Köppen–Geiger climate classification system
  2. Trewartha climate classification
  3. Holdridge life zones

‪While not strictly biomes or ecoregions, these approaches lend themselves to algorithmic definitions that can act as proxies.‬

@oubiwann
Copy link
Member Author

@oubiwann
Copy link
Member Author

oubiwann commented Dec 3, 2019

Working on several biome possibilities in the following spreadsheet:

Current state of last sheet (modification of Holdridge):
Screen Shot 2019-12-03 at 1 39 13 AM

@oubiwann
Copy link
Member Author

oubiwann commented Dec 3, 2019

Updated for more high temps (also, divided for easy color reference):
Screen Shot 2019-12-03 at 2 53 02 AM

@oubiwann
Copy link
Member Author

oubiwann commented Dec 4, 2019

Okay, so after days of exploring the possibility of using roaring bitmaps, B-trees, R-trees, etc., and weighing the size of the biome lookup data as well as the fairly expensive operations that will be done at the same time (reading two images, pixel at a time), here's what I think I'm going to do:

  • Put the axes labels in the spreadsheet into a sorted map.
  • Create a 2d lookup table/matrix with these two sorted mapsl
  • For all inputs, use the Clojure AVL to rank the temp and precip coord to find their nearest neighbor/axis label.
  • Out of range scenarios: anything higher than 369, set to 369; lower than 273, set to 273.

@oubiwann
Copy link
Member Author

oubiwann commented Dec 5, 2019

Good progress ... ended up doing a few bits differently, but mostly as described above.

Here's an ANSI color display of the biome colors:
Screen Shot 2019-12-04 at 11 52 00 PM

Almost done, now ...

@oubiwann
Copy link
Member Author

oubiwann commented Dec 5, 2019

Also added a legend function:
Screen Shot 2019-12-05 at 1 41 27 AM

@oubiwann
Copy link
Member Author

oubiwann commented Dec 5, 2019

I've generated a few biome images, and it's not coming out as planned ... then I took a closer look at the temp ranges I had set up, and those aren't going to work at all ... temperature will need to be reworked :-/

@oubiwann
Copy link
Member Author

oubiwann commented Dec 6, 2019

Here's a current, buggy biome map:

001-mercator-offset-biomes

Possible issues with temp that need to be investigated/addressed:

001-mercator-offset-temperature

  • The function scales/coord->temperature which operates in degrees Kelvin is sometimes returning 0; the lowest temp we should ever see is in the low 200s, so this is a bug
  • The high-temp red zones -- and some orange/yellow zones -- are being treated as polar; I suspect this is related to the bug above
  • The low global average temperature is too low for an Earth-type low average
  • The high global average is too high for an Earth-type high average
  • The distribution of temperatures is too linear, and this results in too much of the temperature range being in near-polar bio zone; things to try:
    • generating a normal distribution for the temp ranges
    • using a trigonometric function (e.g., sin) for the temp ranges
    • something else?

Possible issues with precip that may need to be investigated/addressed, also:

001-mercator-offset-precipitation

  • It seems that there's too much land area in the higher ranges of precipitation; many examples you can find in the literature reference a doubling of precip at every axis tick, so we should explore a logrithmic scale for precips
  • Such an approach may nee to be modulated, though ... since a non-rigorous effort was made to do this when creating the precipitation map; for the map as-is, a straight-up log scale may result in too dry a planet ...

@oubiwann
Copy link
Member Author

oubiwann commented Dec 6, 2019

To solve the issue around the buggy 0 temps, reds, and orange/yellows, here's where we can start:

  • Read the temp color map, building a list of all unique colors
  • Save these to disk; we'll use for testing from now on
  • These should map, 1-to-1, to the hex color list file
    • Convert them from RGB to hex
    • Build a set of both
    • Perform a difference, and make sure that set is the empty set
  • Run this list of unique values through the temp conversion functions
  • Examine the output, looking for anomalies

This has been done, and it seems that one of the problems is that the map image that is index by Gimp using a 42-color palette (the temperature map) actually has 47 colors in it. This was discovered by reading the file and its pixels from Clojure-wrapped Java and stuffing all the color values into a set.

Currently looking at the possibility of generating a palette from the 47-color image, and just using that ...

@oubiwann
Copy link
Member Author

oubiwann commented Dec 7, 2019

Setup for checking colors:

(def ref-hex-vals (->> (scales/temperature-colors)
                       (map util/color-map->hex)
                       (into #{})))
(def im (biome-tmp/read-adjusted-temperature))
(def rgb-vals (into #{} (map-io/all-pixels im)))
(def hex-vals (->> rgb-vals
                   (mapv util/rgb-pixel->hex)
                   (into #{})))
(set/difference hex-vals ref-hex-vals)
#{"0x00b1dd"
  "0x00d6b9"
  "0x0c38e0"
  "0x0ed6a6"
  "0x155cdc"
  "0x1f4edc"
  "0x2c9aca"
  "0x3000df"
  "0x8900e4"
  "0x94d662"
  "0x9953dc"
  "0xd9a300"}
(set/difference ref-hex-vals hex-vals)
#{"0x00b0dd" "0x00d6b8" "0x165cdc" "0x1b37e0" "0x7700e2" "0x8a00e5" "0xdaa200"}

So, in both sets, we have colors that are not in the other ... time to re-index the image, I think ...

Update: that didn't work; going to try using the colors found in the actual image as the basis for the palette instead.

@oubiwann
Copy link
Member Author

oubiwann commented Dec 9, 2019

Fantastic! Color palette problem fixed:

001-mercator-offset-biomes

Next up, need to do a better job of setting the temperature distribution on the planet ...

@oubiwann
Copy link
Member Author

I've been working on creating new temperature scales to experiment with; after a significant refactor that will make that easier, I've started tweaking the temperature scales, color ranges, etc., as I do trial-regenerations of the biomes. The temperature work is flexible enough to experiment with nicely, so will work on doing the same with precipitation.

@oubiwann
Copy link
Member Author

Precipitation has now been brought into the new abstraction, so I will now be able to create an exponential scale for precipitation ... fingers crossed, that will fix many of the biome-generation issues.

@oubiwann
Copy link
Member Author

Sadly, none of the new scales fixes the issues with the biome-generation; here's the latest:

biomes

@oubiwann
Copy link
Member Author

oubiwann commented Dec 25, 2019

As a result of the problems with biome generation, I've created a new biome reporting namespace that reads temp and precip data from images and prints out stats on these; it also uses the same function that the biome-image-generator uses, but just the data and there's a function that prints that out too.

From the output of the temp and precip stats, we can see that those look good (data/stats as expected):

Screen Shot 2019-12-25 at 11 41 51 AM

Screen Shot 2019-12-25 at 11 42 55 AM

But when the biome stats are gathered, we see the same issue as is in the image:

Screen Shot 2019-12-25 at 12 13 17 PM

As we can see, all of the expected temperatures are represented, but the precipitation seems not only low but actually wrong: while the stats show zero precip (for everything!), there is no occurrence of zero precip in the actual data; the lowest precip in our data is 250 mm/yr. There's a chance that some or all of our problems here are related to a bad munging of biome precip data ...

@oubiwann
Copy link
Member Author

oubiwann commented Dec 25, 2019

Huh:

 (set (for [x (range (map-io/width precip-im)) 
            y (range (map-io/height precip-im))] 
        (scales/coord->precipitation biome/ps precip-im x y)))
#{0}

All lookups result in zero.

Yet the reporting functionality does get proper, non-zero lookups (see the output above). Ah, the report uses a different function for lookups:

(scales/precipitation-amount biome/ps (:color-map datum))

Getting closer ...

@oubiwann
Copy link
Member Author

oubiwann commented Dec 25, 2019

Excerpt from running the above set/for loop with TRACE enabled:

Screen Shot 2019-12-25 at 12 33 45 PM

So, it's getting a color map, but it's not able to get a precipitation from it ...

(def precip-stats (reporter/get-precip-stats precip-im))
(def colors (map :color-map (:freqs precip-stats)))
(sort-by :blue colors)
({:blue 27 :green 95 :red 2}
 {:blue 32 :green 106 :red 0}
 {:blue 40 :green 119 :red 0}
 {:blue 46 :green 126 :red 0}
 {:blue 52 :green 132 :red 6}
 {:blue 56 :green 136 :red 18}
 {:blue 63 :green 145 :red 33}
 {:blue 71 :green 152 :red 44}
 {:blue 77 :green 159 :red 52}
 {:blue 77 :green 167 :red 64}
 {:blue 84 :green 173 :red 79}
 {:blue 90 :green 180 :red 89}
 {:blue 97 :green 187 :red 97}
 {:blue 105 :green 195 :red 113}
 {:blue 112 :green 207 :red 136}
 {:blue 135 :green 216 :red 157}
 {:blue 159 :green 224 :red 177}
 {:blue 170 :green 228 :red 187}
 {:blue 180 :green 230 :red 195}
 {:blue 182 :green 231 :red 203}
 {:blue 184 :green 231 :red 211}
 {:blue 186 :green 232 :red 219}
 {:blue 189 :green 232 :red 227}
 {:blue 190 :green 232 :red 234}
 {:blue 193 :green 232 :red 242}
 {:blue 205 :green 237 :red 252}
 {:blue 214 :green 240 :red 253}
 {:blue 223 :green 242 :red 254}
 {:blue 232 :green 247 :red 251}
 {:blue 242 :green 251 :red 253}
 {:blue 251 :green 255 :red 252})
(count colors)
31
(def generated (map util/rgb-pixel->color-map 
                    (set (for [x (range (map-io/width precip-im)) 
                               y (range (map-io/height precip-im))] 
                           (map-io/rgb precip-im x y))))))
(clojure.set/difference (set colors) generated)
#{}

And looking at the log output, we can see that these colors are present in the maps above.

Hrm. Curiouser and curiouser ...

@oubiwann
Copy link
Member Author

Ah, but the colors are in there! Seems that the lookup isn't working due to the sometimes differing order of the map k/v pairs!

@oubiwann
Copy link
Member Author

Nope, that was a red-herring. Using sorted maps didn't change anything ... manual checks still failed.

Let's use some data that failed a lookup in the logged output:

(def x 239)
(def y 152)
(def rgb -3938636)
(def cm (sorted-map :blue 180 :green 230 :red 195))

Do checks against our data:

[(= rgb (map-io/rgb precip-im x y)) (= cm (util/rgb-pixel->color-map rgb))]
[true true]

Confirm that it fails:

(scales/coord->precipitation biome/ps precip-im x y)
2019-12-25T13:36:02.054 [nRepl-session-a88c8c9c-78ea-4c12-b320-9d802be5d225] DEBUG hxgm30.map.scales.precipitation:71 - Getting precipitation for [239, 152] ...
2019-12-25T13:36:02.054 [nRepl-session-a88c8c9c-78ea-4c12-b320-9d802be5d225] TRACE hxgm30.map.scales.precipitation:73 - RGB pixel: -3938636
2019-12-25T13:36:02.054 [nRepl-session-a88c8c9c-78ea-4c12-b320-9d802be5d225] TRACE hxgm30.map.scales.precipitation:75 - Color map: {:blue 180, :green 230, :red 195}
2019-12-25T13:36:02.054 [nRepl-session-a88c8c9c-78ea-4c12-b320-9d802be5d225] TRACE hxgm30.map.scales.precipitation:66 - Precipitation amount: nil
2019-12-25T13:36:02.055 [nRepl-session-a88c8c9c-78ea-4c12-b320-9d802be5d225] TRACE hxgm30.map.util:179 - Getting mean for nil
2019-12-25T13:36:02.055 [nRepl-session-a88c8c9c-78ea-4c12-b320-9d802be5d225] TRACE hxgm30.map.scales.precipitation:77 - Precipitation: 0
0

Yet direct lookups are fine:

(scales/precipitation-amount biome/ps cm)
2019-12-25T13:36:37.490 [nRepl-session-a88c8c9c-78ea-4c12-b320-9d802be5d225] TRACE hxgm30.map.scales.precipitation:66 - Precipitation amount: [6500.0 7000.0]
2019-12-25T13:36:37.490 [nRepl-session-a88c8c9c-78ea-4c12-b320-9d802be5d225] TRACE hxgm30.map.util:179 - Getting mean for [6500.0 7000.0]
6750

And the same logic that fails in coord->precipitation actually works when performed manually:

(scales/precipitation-amount biome/ps (util/rgb-pixel->color-map -3938636))
2019-12-25T13:38:27.627 [nRepl-session-a88c8c9c-78ea-4c12-b320-9d802be5d225] TRACE hxgm30.map.scales.precipitation:66 - Precipitation amount: [6500.0 7000.0]
2019-12-25T13:38:27.628 [nRepl-session-a88c8c9c-78ea-4c12-b320-9d802be5d225] TRACE hxgm30.map.util:179 - Getting mean for [6500.0 7000.0]
6750

Ah-HA!!!

The threading macro ... that changed between temp and precip; setting the precip code to use the same threading macro ->> fixed the problem.

Now we're back to the rain-saturated biome map, and can experiment with the scales.

@oubiwann
Copy link
Member Author

oubiwann commented Dec 25, 2019

Horray! Just applied some fixes to the exponential precipitation scales, and have generated some biome images that are starting to look pretty good:

biomes

That biome looks like the following, once oceans and altitude shading are overlaid:

001-mercator-bump-sea-offset-biomes-medium

This still isn't quite right, since with the amount of rainfall hitting that large, oceanic island, there should be a lot of rainforest there. This leads me to believe the current combinations of temperature and precipitation scales still need tweaking.

@oubiwann
Copy link
Member Author

Okay, I think I've got something I can be happy with:

biomes

With the oceans, that gives this:

001-mercator-offset-biome-medium

This biome iteration used the following:

(def ts (scales/new-scale :temperature :gaussian 0.4 -0.1))
(def ps (scales/new-scale :precipitation :exponential 5))

After a lot of trial and error, the Gaussian normal really shone for the temperature maps. The first parameter is sigma2, and the second parameter is mu.

Legends for the above:

Screen Shot 2019-12-27 at 1 56 12 AM

Screen Shot 2019-12-27 at 1 58 29 AM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant