Feature: Import real-world town data via JSON file #10409

2TallTyler · 2023-01-25T01:29:22Z

Motivation / Problem

Creating real-world scenarios in Scenario Editor is slow and time-consuming.

Aligning towns to the correct location without a real map underlay is tricky and involves a lot of guessing.
NewGRFs are hard-coded to the scenario file. While many can be loaded after building the scenario, house NewGRFs cannot be changed without deleting all towns, which is most of the work involved in creating a typical scenario.
Changing map size requires starting from scratch.

Description

The Town Generation GUI in Scenario Editor now lets you load towns from a JSON file, each containing the town's name, location, target OpenTTD population, and whether it is marked as a city in the game.

The intended way of generating this information is in an image editor, with the accompanying heightmap and a labeled map aligned with each other in different layers. The pixel coordinates of the town become (with some easy conversion) the tile coordinates of the town in OpenTTD.

The benefits of this approach are:

Placing towns accurately is much easier using a map underlay such as OpenStreetMap to match town locations with the corresponding heightmap.
Storing town data in a JSON file instead of as an OpenTTD Scenario (.scn) doesn't require storing NewGRFs, particularly the NewGRF house set. You can load the data with whatever house set you choose.
Town coordinates are scaled by the map size, so you can choose the size that works for you.

This standardized format is an API which allows future tools to generate this data and feed it to OpenTTD.

You can download a sample heightmap and town data set from the .zip below. The heightmap is placed as usual in OpenTTD\scenario\heightmap and the town data goes in OpenTTD\scenario\towndata (which should be created automatically).

example_data.zip

To Do

Add a progress bar for generating towns. If there are many towns and/or the population is large, it can take several seconds to complete.
Separate the placing and growing of towns, to place all towns before expanding. This will avoid large towns blocking nearby towns from being placed, in a dense urban area.

For a later PR

Add the ability to save town data from an existing game, to allow recreating a favorite game or scenario with different NewGRFs, using Scenario Editor.

Planned for separate PRs

A few notes on my future plans to head off concerns or scope creep suggestions. 😉

Redesign the Town Generation GUI ([Bug]: Useless buttons when founding town in game #10287). It was bad enough before I added another button.
Add features for importing (as a PNG) trees, desert tiles, and objects, from their own GUIs. (This is why town data importing is via the Town Generation GUI instead of the save/load pull-down like heightmaps.)

Questions

Should BaNaNaS support uploading and downloading this data (maybe bundled with the heightmap as a .tar)? I want to consider compatibility with future data layers like trees and desert (not objects though, that's a manual-only tool). I would need help with this. 🙂
It would also be possible to save existing OpenTTD maps and scenario data to this format, fulfilling the reproducible map generation which is requested every so often. But I think this is somewhat out of scope for this PR and could be added later. Thoughts?

Checklist for review

Some things are not automated, and forgotten often. This list is a reminder for the reviewers.

The bug fix is important enough to be backported? (label: 'backport requested')
This PR touches english.txt or translations? Check the guidelines
This PR affects the save game format? (label 'savegame upgrade')
This PR affects the GS/AI API? (label 'needs review: Script API')
- ai_changelog.hpp, gs_changelog.hpp need updating.
- The compatibility wrappers (compat_*.nut) need updating.
This PR affects the NewGRF API? (label 'needs review: NewGRF')
- newgrf_debug_data.h may need updating.
- PR must be added to API tracker

src/misc/json.hpp

src/3rdparty/json11/json11.cpp

LC-Zorg · 2023-01-27T00:17:17Z

For scenario creators this is a really great feature. :) Thanks to this, the scenario of the map of Poland, the Visegrad group (Poland, Czech Republic, Slovakia, Hungary) and Europe, which used to be also on the server, was created. The first scenario already had a lot of updates, including changes to industries sets and scripts, which required the map to be regenerated from scratch. Without this feature, it would hardly make sense. Here I would like to point out that a version of the game was used to generate the map (added the aforementioned McZapkie's repeatable map generation tools) which also allowed importing rivers, objects (stone and transmitters as borders) and enterprises - features that would be equally useful. I admit that although I helped a lot in refining these maps, I've never used this version of the game (needs to be compiled) and I don't know exactly how it works. I know that rivers and objects can be imported from an image file and all functions are only available from the console level so it's not very convenient for the average user.

Should BaNaNaS support uploading and downloading this data (maybe bundled with the heightmap as a .tar)? I want to consider compatibility with future data layers like trees and desert (not objects though, that's a manual-only tool). I would need help with this.

I think definitely yes, but maybe it would be worth adding a new format?

Heightmap - the height map itself
Map - height map with other data or layers added (city locations, rivers, water depth, deserts, snow, trees, roads and more), the player still has the option to choose his own industrial set, roads, buildings, vehicles and script; allowing for the collection of, for example, the water layer itself or cities would not make sense without making available the base it refers to
Scenario - fully ready map

In the case of the Map and the Scenario, it would be good if it was possible to export individual layers to graphic files (water, trees, etc.) or text files (cities, enterprises)

The ability to use more layers could be added gradually. Although I write this as a theoretician, it seems to me that the most important thing is to define the framework of the basis on which all this could be based.

For the future:
As for deserts and snow, I think it would be worthwhile to go even further and make maps with many different substrates. For example, a map of Europe from snowy Scandinavia to the deserts of northern Africa (look at tt-distas server with map of Europe). The same with the map of the United States itself, where desert or rocky areas can connect directly to snow-covered areas. Snow could be a separate layer, where different colors on png file would correspond to seasonal changes.

When it comes to rivers and lakes, it took me over 30 hours to refine them on the Visegrad map. I think being able to save this layer, outside of cities, would be most helpful.

Redesign the Town Generation GUI. It was bad enough before I added another button.

I'm not sure if it's worth changing it any more significantly - I wrote a bit more here #10287. :)

src/genworld.cpp

TrueBrain · 2023-09-10T18:02:45Z

src/genworld.cpp

+		do {
+			Command<CMD_EXPAND_TOWN>::Post(t->index, 0);
+			if (tries-- == 0) break;
+		} while (t->cache.population < town.population);


This is a bit weird .. only 500 tries? Why not 500 failures? Feels a bit random, 500.

Why not 500 tries? This is to expand the town to the target population specified by the player in the JSON file, so we need some way to break out of the do/while loop in case their target population is not reachable. (Which is extremely likely, since some players will inevitably try to match real-world city populations of 1M+, which doesn't work in OpenTTD...)

Alternate suggestions welcome. 😃

src/genworld.h

TrueBrain · 2023-09-10T18:06:09Z

src/fileio.cpp

@@ -42,6 +42,7 @@ static const char * const _subdirs[] = {
 	"save" PATHSEP "autosave" PATHSEP,
 	"scenario" PATHSEP,
 	"scenario" PATHSEP "heightmap" PATHSEP,
+	"scenario" PATHSEP "towndata" PATHSEP,


Just as a FYI, if you want to be able to load heightmaps + towndata, they can't be in different folders. How the content system works, is that 1 content piece needs everything in a single folder.

But I also like your idea of having this for SE-only, that people need to load towndata on top of their scenario / heightmap. But just the headsup.

Ah, thanks for noting this. Something to consider as we move forward with this and consider bundling town data with heightmaps or other info.

Not who this affects considerations, but I would think that town placement data is only useful for a specific heightmap. A scenario can be considered a heightmap and town data pre-combined.

My rationale for doing towns and heightmaps separately instead of bundling them in a .tar or similar, is that the heightmap import often doesn't go perfectly the first try. The max height is wrong, or some coasts need cleanup, etc. Loading towns takes a lot of time, so it would be frustrating to force them to happen simultaniously.

For now, this format is for scenario creators, not a full "reproducible scenario" format for use by the average player.

That still doesn't answer the question of where to put the files. I guess we could always move them later.

docs/importing_town_data.md

PeterN · 2023-11-01T09:02:20Z

src/fileio.cpp

@@ -42,6 +42,7 @@ static const char * const _subdirs[] = {
 	"save" PATHSEP "autosave" PATHSEP,
 	"scenario" PATHSEP,
 	"scenario" PATHSEP "heightmap" PATHSEP,
+	"scenario" PATHSEP "towndata" PATHSEP,


Not who this affects considerations, but I would think that town placement data is only useful for a specific heightmap. A scenario can be considered a heightmap and town data pre-combined.

src/fios_gui.cpp

src/genworld.cpp

PeterN · 2023-11-01T21:39:06Z

src/genworld.cpp

+
+	/* Add a trailing \0 to mark the end of the string */
+	text = ReallocT(text, filesize + 1);
+	text[filesize] = '\0';


Pretty sure you should be able to read directly into a std::string (resized to filesize), instead of affing with C-style memory allocation and null-termination.

Yes, I just can't figure out how... 😕

src/genworld.h

2TallTyler · 2024-01-23T01:00:49Z

Updated and rebased. There are still outstanding review comments which I need help with.

2TallTyler added the work: still in progress (draft) This Pull Request is a draft label Jan 25, 2023

2TallTyler force-pushed the geodata branch from 9c0892b to 0e6a06f Compare January 25, 2023 02:35

glx22 reviewed Jan 25, 2023

View reviewed changes

src/misc/json.hpp Outdated Show resolved Hide resolved

github-advanced-security bot found potential problems Jan 25, 2023

View reviewed changes

This comment was marked as outdated.

Sign in to view

2TallTyler force-pushed the geodata branch from 0e6a06f to a308b6a Compare January 25, 2023 18:31

This comment was marked as outdated.

Sign in to view

2TallTyler mentioned this pull request May 25, 2023

[Bug]: Useless buttons when founding town in game #10287

Closed

TrueBrain reviewed Sep 10, 2023

View reviewed changes

TrueBrain added the candidate: yes This Pull Request is a candidate for being merged label Sep 13, 2023

2TallTyler force-pushed the geodata branch 2 times, most recently from 87bbe0b to 0cea5d4 Compare October 16, 2023 16:06

2TallTyler mentioned this pull request Oct 16, 2023

[Bug]: Cannot expand towns in Scenario Editor when town road construction disallowed #11374

Closed

2TallTyler force-pushed the geodata branch 3 times, most recently from 5ed5625 to a5c3662 Compare October 31, 2023 23:17

PeterN reviewed Nov 1, 2023

View reviewed changes

2TallTyler force-pushed the geodata branch from a5c3662 to 7895396 Compare November 2, 2023 15:17

TrueBrain added this to the 14.0 milestone Nov 29, 2023

2TallTyler force-pushed the geodata branch from 7895396 to a353e5e Compare January 23, 2024 01:00

2TallTyler removed the work: still in progress (draft) This Pull Request is a draft label Jan 28, 2024

2TallTyler marked this pull request as ready for review January 28, 2024 03:22

2TallTyler added 2 commits January 28, 2024 09:36

Doc: Add reference and tutorial for external town data

1add818

Feature: Import real-world town data via JSON file

9fcfdc5

2TallTyler force-pushed the geodata branch from a353e5e to 9fcfdc5 Compare January 28, 2024 14:46

TrueBrain removed this from the 14.0 milestone Feb 1, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature: Import real-world town data via JSON file #10409

Feature: Import real-world town data via JSON file #10409

2TallTyler commented Jan 25, 2023 •

edited

This comment was marked as outdated.

LC-Zorg commented Jan 27, 2023

This comment was marked as outdated.

This comment was marked as outdated.

TrueBrain Sep 10, 2023

2TallTyler Oct 16, 2023

TrueBrain Sep 10, 2023

2TallTyler Oct 16, 2023

PeterN Nov 1, 2023

2TallTyler Nov 2, 2023

PeterN Nov 1, 2023

PeterN Nov 1, 2023

2TallTyler Nov 2, 2023

2TallTyler commented Jan 23, 2024

Feature: Import real-world town data via JSON file #10409

Are you sure you want to change the base?

Feature: Import real-world town data via JSON file #10409

Conversation

2TallTyler commented Jan 25, 2023 • edited

Motivation / Problem

Description

To Do

For a later PR

Planned for separate PRs

Questions

Checklist for review

This comment was marked as outdated.

LC-Zorg commented Jan 27, 2023

This comment was marked as outdated.

This comment was marked as outdated.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

2TallTyler commented Jan 23, 2024

2TallTyler commented Jan 25, 2023 •

edited