Repository navigation
Calibration and validation
Calibration and validation test whether a land use model reproduces observed land use. Calibration sets the parameters of the model on observed data, mostly the weights of the suitability factors. Validation then compares the result with data that the calibration did not use, usually a later year.
A simulation can go wrong in the amount of land and in its location. The amounts mostly come from regional demand outside the model, so a test of the model itself usually looks at location. It allocates the amounts that were actually observed in the test year, and any difference with the observed map then comes from the suitability and the allocation method.
Loonen & Koomen (2009) calibrated and validated both allocation methods of Land Use Scanner on a 100 m grid with nine land use types (Loonen, Koomen & Kuijpers-Linde, 2007):
- Suitability comes from a multinomial logistic regression on a random half of the 3.3 million cells, and the other half serves as an independent test. One specification uses only the surrounding land use, and the other adds accessibility, spatial policy and underground infrastructure.
- Calibration reproduces the land use of the base year 1993, and validation simulates 2000 with the relations found for 1993.
- The comparison is cell by cell and per land use type, because an overall measure would mainly reflect agriculture as the largest type.
Both methods gave sensible and very comparable patterns. Land use in the surrounding cells turned out to be important for most land use types, which points to spatial autocorrelation.
The other tests in the Literature include the validation of the LUMOS land use modelling system (de Nijs, Boersma & Koomen) and the description and validation of EU-ClueScanner100m (Koomen, Diogo, Hilferink & van der Beek, 2010).
- Most land does not change within a few years, so a high agreement between simulated and observed maps can mainly reflect land that stayed the same. Changed land is the hard part of a simulation.
- A relation estimated on one year and tested over seven years says little about a period of decades, when other factors may become more important.
- The aim of a scenario study is to explore possible futures rather than to replicate the past. The suitability weights of a scenario express a storyline and cannot be validated with observations.
- There are no standards. A review of 114 land-change model applications found that about a third reported no validation at all, and that most of the others looked at locational accuracy rather than at the pattern of land use (van Vliet et al., 2016).
The Land Use Scanner demo is not calibrated. Its users set the weights in the scenario scripts themselves. Its evaluation indicators show how well the claims are met and what changes, but do not compare the result with an observed map.
- Loonen, W., & Koomen, E. (2009). Calibration and validation of the Land Use Scanner allocation algorithms. PBL publication 550026002. Bilthoven: Netherlands Environmental Assessment Agency.
- Loonen, W., Koomen, E., & Kuijpers-Linde, M. (2007). Calibrating and validating the Land Use Scanner algorithms. 15th European Colloquium on Theoretical and Quantitative Geography, Montreux.
- van Vliet, J., Bregt, A. K., Brown, D. G., van Delden, H., Heckbert, S., & Verburg, P. H. (2016). A review of current calibration and validation practices in land-change modeling. Environmental Modelling & Software, 82, 174-182.
