Blueprint overhaul plan

[myk002]

Design doc and tracking bug for the blueprint plugin overhaul. Public discussion in the blueprint thread. This text covers my ideas for:

• Objectives
• Use cases
• Features and commandline API
• GUI design
• Architecture and algorithm
• Project plan

Objectives

The point of this overhaul is to make blueprint a more productive member of the quickfort ecosystem. Blueprint is the easiest way to create useful, valid quickfort blueprints, especially for people new to quickfort. However, there are a lot of opportunities to make blueprints easier to create, edit, and play back. There are also many ways to improve the quality of the generated blueprints via capturing more map information.

Here are the goals in list form, with tags that we can use later to categorize the project plan tasks:

1. [Reliability] Ensure the blueprint -> quickfort workflow works reliably
2. [Generate] Make it easier to generate usable blueprints from a map
3. [Edit] Make it easier to edit generated blueprints
4. [Playback] Make use of DFHack quickfort usability features to make the blueprints easier to play back
5. [Completeness] Capture as much map information as possible in the generated blueprints

Use cases

There are two major groups of users that this work is targeting:

1. Users who want to generate a blueprint and use it with quickfort without having to understand anything about the format
2. Users who want to use generated blueprints as a starting point for creating more advanced custom blueprints

I am catering the default settings to members of group 1, with most of the non-default options intended for members of group 2.

For group 1, the expected workflow is:

1. run blueprint gui or gui/blueprint to generate a minimal .csv blueprint file
2. run quickfort gui or quickfort run at some later time to apply it

I want everything to be as simple as possible for this group. Quickfort has the capability to display messages after a blueprint is run, so we can autogenerate a walkthrough for replaying the blueprints later.

Group 2 will use a greater variety of options and spend time editing the blueprints before replaying them with quickfort. The features that make blueprints easier to read, understand, and edit are for this group. Group 2's expected workflow is:

1. run blueprint gui or gui/blueprint, fiddle with options, and generate pretty-printed .csv or .xlsx files
2. edit the files in a text editor or spreadsheet program
3. save/export the modified blueprints
4. run them through quickfort

I expect members of group 2 to make iterative changes to the modified blueprints, either via direct editing or by using blueprint to generate blueprints for specific map areas, which they will then use to overwrite portions of their modified blueprints.

Features and commandline API

This section is written in the format of DFHack plugin documentation since we'd have to write that later anyway : )

note: I haven't checked it for formatting bugs yet. Also, this text assumes all features are already implemented. In reality, I plan to add bits of help text incrementally as the related features are completed.

blueprint
=========
The ``blueprint`` command records the structure of a portion of your fortress in
a blueprint file that you (or anyone else) can later play back with `quickfort`.

Blueprints are ``.csv`` or ``.xlsx`` files created in the ``blueprints``
subdirectory of your DF folder. The map area to turn into a blueprint is either
selected interactively with the `blueprint gui` command or, if the gui is not
used, starts at the active cursor location and extends right and down for the
requested width and height.

Usage:

    blueprint <width> <height> [<depth>] [<name> [<phases>]] [<options>]
    blueprint gui [<name> [<phases>]] [<options>]

Examples:

blueprint gui
    Runs `gui/blueprint`, the interactive blueprint frontend, where all
    configuration for a ``blueprint`` command can be set visually and
    interactively.

blueprint gui bedrooms --format pretty
    Starts `gui/blueprint` with some configuration options already set to custom
    values.

blueprint 30 40 bedrooms
    Generates blueprints for an area 30 tiles wide by 40 tiles tall, starting
    from the active cursor on the current z-level. Output is written to the
    ``bedrooms.csv`` file in the ``blueprints`` directory.

blueprint 59 63 -15 --splitby level --format xlsx --playback_start "30;32;central stairs"
    Generates blueprints for an area 59 tiles wide by 63 tiles tall for
    15 z-levels, starting from the active cursor and going down the z-levels.
    Blueprints are written to ``blueprints/blueprint_level1.xlsx`` through
    ``blueprints/blueprint_level15.xlsx``. When the blueprints are run with
    quickfort, the cursor should be placed 29 tiles in and 31 tiles down from
    where the cursor was when the blueprints were created (that is, in row 30
    and column 32 of the blueprint area. the comment indicates that the
    cursor should be placed where you want the central stairs of the fortress to be).

Positional Parameters:

:width:   Width of the area (in tiles) to translate.
:height:  Height of the area (in tiles) to translate.
:depth:   Number of z-levels to translate. Positive numbers go *up* from the
          cursor and negative numbers go *down*. Defaults to 1 if not specified,
          indicating that the blueprint should only include the current z-level.
:name:    Base name for blueprint files created in the ``blueprints`` directory.
          If no name is specified, "blueprint" is used by default. The string
          must contain some characters other than numbers so the name won't be
          confused with the optional ``depth`` parameter.

Phases:

If you want to generate blueprints only for specific phases, add their names to
the commandline, anywhere after the blueprint base name. You can list multiple
phases; just separate them with a space.

:dig:    Generate quickfort ``#dig`` blueprints for digging through natural wall tiles.
:carve:  Generate quickfort ``#dig`` blueprints for smoothing and engraving, including carving tracks.
:construct:  Generate quickfort ``#build`` blueprints specifically for
         constructions, like walls and floors. Constructions and buildings are
         written to separate blueprints so walls can be built before doors and
         floors can be built before the furniture that sits on top of them.
:build:  Generate quickfort ``#build`` blueprints for building non-construction
         buildings.
:place:  Generate quickfort ``#place`` blueprints for placing stockpiles.
:zone:   Generate quickfort ``#zone`` blueprints for designating zones.
:query: Generate quickfort ``#query`` blueprints for non-room configuration.
         Query commands that can be run immediately after the previous phases
         without waiting for anything to be constructed (like stockpile
         configuration) go in this phase.
:rooms:  Generate quickfort ``#query`` blueprints for configuring rooms and
         other query commands that can't be run until after furniture is placed.

If no phases are specified, blueprint will autodetect which blueprints should
be generated. For example, if there are no zones defined in the area, no
blueprint will be created for the ``zone`` phase. If a phase was explicitly requested,
a blueprint for that phase will be generated, even if there is nothing in it.

Options:

:-c, --cursor <x>,<y>,<z>:
    Use the specified map coordinates instead of the current cursor position for
    the boundary of the blueprint range. If this option is specified, then an
    active cursor is not necessary.
:-f, --format <format>:
    Select the output format of the generated files. See the ``Output Formats``
    section below for options. If not specified, the output format defaults to
    "minimal", which will produce a small ``.csv`` file.
:-h, --help:
    Show command help text.
:--nometa:
    `Meta blueprints <quickfort-meta>` let you apply all blueprints that can be
    replayed at the same time (without unpausing the game) with a single
    command. This usually reduces the number of quickfort commands you need to
    run to rebuild your fort from about 6 to 2 or 3. If you would rather just have the
    low-level blueprints, this flag will prevent meta blueprints from being generated
    and any low-level blueprints from being `hidden <quickfort-hidden>` from the
    ``quickfort list`` command.
:-s, --playback_start <col number>;<row number>;<descriptive comment>:
    Specify the column and row (relative to the upper-left corner of the blueprint)
    where the player should put the cursor when the blueprint is played back
    with quickfort, in `quickfort start marker <quickfort-start>` format. E.g.:
    ``"10;10;central stairs"``. Note that if there is a space in the comment, you
    need to surround the entire string in double quotes.
:--smooth <opt>:
    How much smoothing to capture in the 'carve' phase. Two options: 'fortifications-only' is the default and will only smooth wall tiles that will become fortifications. 'all' captures all smoothed tiles. Note that smoothing all tiles before continuing on with other phases will significantly slow down your progress when playing back the blueprints.
:-t, --splitby <strategy>:
    Split blueprints into multiple files. See the ``Splitting output into
    multiple files`` section below for details. If not specified, defaults to
    "none", which will create a standard quickfort
    `multi-blueprint <quickfort-packaging>` file.

Output formats:

Here are the values that can be passed to the ``--format`` flag:

:minimal:
    Creates ``.csv`` files with minimal file size. This is the default.
:pretty:
    Makes the blueprints in the ``.csv`` files easier to read and edit with a text
    editor by adding extra spacing and alignment markers.
:xlsx:
    Creates a binary ``.xlsx`` file with each blueprint on a separate worksheet.
    This option is best for viewing and editing the blueprints in a spreadsheet
    application like Microsoft Excel or `Google Sheets <https://sheets.new>`__.

Splitting output into multiple files:

Here are the values that can be passed to the ``--splitby`` flag. The "ext" in
the filename formats below represents either "csv" or "xlsx", depending on the
requested output format.

:none:
    Writes all blueprints into a single file. This is the standard format for
    quickfort fortress blueprint bundles and is the default.
:level:
    Creates one file of blueprints per z-level. Filenames will be in the format:
    basename_levelN.ext, where N is the level number, starting from 1 as the
    top-most level.
:group:
    Creates one file per group of blueprints that can be played back at the same
    time (without have to unpause the game and let dwarves fulfill jobs between
    blueprint runs). Filenames will be in the format: basename_phases.ext, where
    phases is a string like "dig" or "build_place_query", depending on which
    phases are included in the group.
:phase:
    Creates a separate file for each individual phase. Implies ``--nometa``
    since meta blueprints can't combine blueprints that are in separate files.

GUI design

Right panel is overwritten with the blueprint interface and all keyboard input
is intercepted. Panel has 28 columns for text. Format:

Blueprint

Create quickfort blueprints
from a live game map.

Select the first corner.


n: name  blueprint

a: Phases  Autodetect

f: format  Minimal text .csv
  File output format

m: meta  On
  Combine blueprints that
  can be replayed together.

s: playback start tile  Unset
  Set where the cursor
  should be positioned when
  replaying the blueprints.

t: split files:  No
  Split blueprints into
  multiple files.

ESC: Back

The game cursor is active and can be moved with the cursor keys. When Enter is
hit, the Select the first corner. line changes to Select the second corner
and the ESC line changes to:

ESC: Cancel selection

After the first corner is selected, a green flashing X appears where the cursor
is. As the cursor is moved, the area from the start position to the current
cursor position will flash with green X characters. The selected dimensions and
area will appear on the gui panel, for example:

Select the second corner.
  10x15x1 (150 tiles)

When the second corner is selected, the gui sends the configuration to the main
blueprint generation routine (as if the blueprint command were invoked directly
with those options). Then a summary dialog is shown detailing which files were created.

• n replaces the current name string with an editable field, pre-populated
  with the current name (or a blank if the current name was the default "blueprint" string).
  Enter commits the new name. ESC dismisses the edit widget
  without changing the name. If files with the given basename are detected in the
  output directory, a warning is displayed, e.g.:

n: name  blueprint
  Warning: may overwrite
  existing files.

• a toggles between Autodetect and Custom, with the following lines shown
  if Custom is set. Each phase toggles between On and Off.

a: Phases  Custom
d: dig        On
D: smooth / carve    On
B: construct  On
b: build      On
p: place      On
z: zone       On
q: query     On
Q: rooms      On

• f cycles across Minimal text .csv, Pretty text .csv, and Binary .xlsx
• m toggles between On and Off
• s starts a workflow where the player can select a map tile to use as the
  playback start position, and is then prompted to enter a string to use as the comment.
  When set, it will look like this:

s: playback start tile  Set
Comment: user-entered description, truncated to fit in panel width
  Set where the cursor...

Hitting s when the start tile has been set will toggle it back to Unset (though the comment will be saved for when the user selects s again to select a different tile).

• t cycles across No, Level, Group, and Phase

Minimal UI workflow

Although setting of the blueprint name is technically optional, I expect the minimal flow to be:

1. hit n, enter a name, hit Enter
2. use cursor or mouse to select starting tile
3. use cursor or mouse to select opposite corner of target area

As soon as the second corner is selected, appropriate blueprints will be generated and the summary dialog is shown.

Architecture and algorithm

Files:

• plugins/blueprint.cpp
  • main loop
  • map state queries
  • i/o
• plugins/lua/blueprint.lua
  • commandline processing
  • support logic (as long as it is not called for every tile)
• scripts/gui/blueprint.lua
  • interactive overlay
  • assembles parameters and invokes blueprint plugin

Here's my attempt at an architecture diagram (thanks, asciiflow.com!)

            ┌───────────────────────────────────┐
            │                                   │
user        │ ┌───────────────┐                 │
invocation  │ │               │                 │
────────────┴─► blueprint.cpp ├────► Disk       │
              │               │                 │
              └───┬───────────┘                 │
                  │commandline processing       │
                  │other low-qps logic          │
              ┌───▼────────────────────────┐    │
              │                            │    │
              │ plugins/lua/blueprint.lua  │    │
              │                            │    │
              └───┬────────────────────────┘    │ assembled
                  │invokes gui script           │ command
                  │if requested on commandline  │
user          ┌───▼────────────────────────┐    │
invocation    │                            │    │
──────────────► scripts/gui/blueprint.lua  ├────┘
              │                            │
              └────────────────────────────┘

We'll also need to extend the xlsxreader plugin to support writing .xlsx files. The xlsxio library already supports writing. We'll just need to wire it up through our xlsxreader plugin (and rename it?) or write a new plugin just for writing.

Overall, I expect the core algorithm to look something like this:

1. scan map for data to put in enabled phases, store in sparse map data structure, tracking max column widths for use in pretty printing
2. output data from groups of maps into appropriate files (according to the --splitby strategy) in the requested format

If sparse maps are too slow, we can use fixed-width pre-allocated buffers with a backup sparse map to pick up any cells that can't fit in the buffer.

Project plan

Each milestone contains a description and component tasks. Tasks items are tagged by the goal they work towards and my initial estimation of task complexity (S/M/L).

Milestone 1 - Fundamentals [done]

Wrap the blueprint entrypoint, implement the basic gui, set up verification
tests, and do other dev preparation tasks. The functional test suite will take
blueprint-generated files, run them through quickfort, run the modified map
back through blueprint, and compare the input to the output. These tests can
be extended incrementally as we add new features to blueprint. It will also act
as an integration test for quickfort, which I haven't otherwise written yet.

I don't want to wait for dig blueprints to be dug out by dwarves, so I'll need something
that reads dig designations and converts them into a "dug" state immediately. Options are
a new plugin/script (modtools/dig-dug? : ) or extending the tiletypes plugin to make target shape
dependent on current dig designation. Tiletypes already has the logic for converting a tile to
a different shape, so adding logic to the plugin instead of creating a new script might be the
best approach.

[Generate M] Support new commandline syntax for blueprint (in lua)
[Generate S] Support --cursor flag
[Generate M] Implement gui/blueprint for current functionality (name, phases)
[Reliability L] insta-dig designated tiles (new plugin or modify tiletypes or ?)
[Reliability L] Functional test suite of: file -> quickfort/buildingplan/dig-dug -> blueprint -> file and compare.

Milestone 2 - Make blueprints easier to use in quickfort [done]

Start making the generated blueprints easier to use in quickfort. Previous
behavior will still be available via flags, but I'll be shifting the defaults.

First, I'll start writing multi-blueprint files instead of one file per phase.
This makes blueprints easier to move around and share. It also makes quickfort
#meta blueprints possible, which will dramatically reduce the toil associated
with reapplying the blueprints later.

Second, I'll introduce the ability to output the files in different formats. The
current output format is somewhere between what I envision the "minimal" and
"pretty" output formats being. Although "pretty" will make .csv files easier to
manually edit, I'm going with "minimal" as the default because it's faster to
read and write, and once I implement the xlsx output format, that will be the
one I will promote as best for editing.

Last, I'll add support for the quickfort start() modeline marker so blueprints
can be played back more naturally.

[Edit M] Support --splitby flag for "none" and "phase" strategies in blueprint and gui/blueprint
[Playback S] Generate quickfort modeline labels from basename and phase
[Edit M] Support --format flag for "minimal" and "pretty" in blueprint and gui/blueprint
[Playback M] Support --playback_start flag in blueprint and gui/blueprint

Milestone 3 - Quality of life improvements [done]

Take a short break from implementing radical changes and fix some of blueprint's
functionality holes.

[Completeness S] Support remaining building types in blueprint
[Completeness S] Support multi-type stockpiles in blueprint
[Generate M] Only write blueprints for phases that have something to do
[Completeness M] Support non-rectangular stockpiles and buildings in blueprint

Milestone 4 - Prep for meta blueprints [done]

Phases! Phases! Phases!

First, add support for carved tracks and zones.

Second, split the build phase. The current build phase combines constructions
and buildings, but this can lead to problems later when you try to attach a door
to an unbuilt wall. This method also misses flooring underneath buildings and
furniture. Splitting off a "construct" phase fixes this.

Finally, split the query phase. The current query phase just designates rooms,
but there's a lot more the query mode can accomplish. In particular, there is a
bifurcation between #query blueprints that can be run immediately after #place
and #build blueprints and #query blueprints that can't be run until buildings
are built and furniture is placed. So redefine "query" as "can be
run immediately" and add a "rooms" phase for query blueprints that must wait. All current functionality would move
to "rooms", so in order for something to be written to the new "query" phase,
we'll take the low hanging fruit of recording building, stockpile, and zone
names in the "query" blueprints. More will be added to these two phases later.

[Completeness S] Support track phase in blueprint and gui/blueprint
[Completeness L] Support zone phase (incl. detailed settings) in blueprint and gui/blueprint
[Playback L] Split construct phase from build phase in blueprint and gui/blueprint
[Playback S] Split query phase into query and rooms phases in blueprint and gui/blueprint
[Completeness S] Record building, stockpile, and zone names in blueprint

Milestone 5 - Meta blueprints [in progress]

Now that we have the phases separated out from the previous milestone, we can
group them properly with meta blueprints:
Group 1: dig
Group 2: construct
Group 3: build, place, zone, query
Group 4: rooms

This reduces group 3 from up to 4 commands down to 1. This is a big usability
improvement.

[Playback M] Generate meta blueprints and support --nometa flag in blueprint and gui/blueprint
[Playback S] Generate modeline messages to indicate when the next blueprint can be run
[Edit M] Support --splitby flag for "group" strategy in blueprint and gui/blueprint

Milestone 6 - Documentation

[Generate/Edit M] Add a section to the quickfort user guide on how to create blueprints with the blueprint plugin and the bits of syntax you need to know to make finishing touches

Gravy

[Completeness M] Record stockpile links (give/take) and links-only setting (as long as target is within translation region in blueprint)
[Completeness L] If stockpile name matches a quickfort stockpile alias, use that alias in the "query" phase
[Completeness L] Record building settings and room configuration
[Edit M] Support --splitby flag for "level" strategy in blueprint and gui/blueprint
[Edit L] Support --format=xlsx flag in blueprint and gui/blueprint
[Edit L] Generate boundary and building shadows in blueprint for --format=pretty and --format=xlsx
[Generate L] Support creating dig blueprints from dig designations instead of real tile shapes

Possible future tasks

[Generate] Ask for feedback on the forums to see if a "cursor follows mouse" mode would be desirable for gui/blueprint
[Completeness L] Save arbitrary stockpile configuration (exported/imported with stockpiles plugin? Will need a way to apply saved data with quickfort.)

[myk002]

@lethosor at a high level, does this plan sound ok to you? we can dig into details in the code reviews.

[lethosor]

Sounds good to me! (at least from a quick read-through.) Honestly, even part 1 would be a significant improvement.

I'll look through the first couple PRs soon. I should have more time this week than I did recently.

I also want to get r2 out due to a couple other unrelated fixes, so if the existing PRs are stable enough for a release, that might be a good cutoff point for r2. (Maybe I should start making proper release branches...)

[myk002]

Some PRs will come in pairs -- one for plugins/blueprint and one for gui/blueprint -- but as long as the pairs are merged in the same release, the code should always be release-ready. I don't know how long it will take me to reach each milestone, so I made sure to plan for arbitrary release cutoffs.

The initial implementation of gui/blueprint in #281 is a bit basic. I have PRs queued up for better usability, but it's also fine to be released in its current state. If people start complaining about its lack of features, I'd just be happy that they're using it : )

[arzyu]

Awesome work!!
It would be better if there are rotation and flip options when applying blueprints.

[myk002]

Agreed. Feel free to request features on the quickfort thread: http://www.bay12forums.com/smf/index.php?topic=176889.msg8306977#new

That particular feature, though, is already pretty high priority on my backlog: https://docs.google.com/document/d/17EjZlGsUOJ6E8WtqkCRZi9BBKjL9L1IdHfkB3Ajqas8/edit?usp=drivesdk

[myk002]

note to self: I seem to have forgotten to expose the description field for the blueprint modeline. I'll need to add that.

edit: oh, I see, the description would potentially need to be different for every phase.. maybe it's better to leave that out. we still have the blueprint name that acts as a unique identifier

[myk002]

I did some work on carved tracks in quickfort, and I realized that blueprint currently completely ignores carved tracks. I added a new "track" phase to support in milestone 4.

Here the code, to remind myself later:

static const char * get_track_str(const char *prefix, df::tiletype tt) {
    TileDirection tdir = tileDirection(tt);

    string dir;
    if (tdir.north) dir += "N";
    if (tdir.south) dir += "S";
    if (tdir.east)  dir += "E";
    if (tdir.west)  dir += "W";

    return cache(prefix + dir);
}

// for floors
        if (tileSpecial(*tt) == tiletype_special::TRACK)
            return get_track_str("track", *tt);
        // fallthrough

// for ramps
        if (tileSpecial(*tt) == tiletype_special::TRACK)
            return get_track_str("trackramp", *tt);

[myk002]

What to do with carved fortifications? They can't be dug in the "dig" phase since the walls have to be smoothed first. Perhaps we should rename the "track" phase to "carve" and have it include tracks and fortifications? We'll also need to add a "smooth" phase between "dig" and "carve". Ugh.

Do we also need options for "minimal smoothing" (i.e. just fortification tiles) vs. "all smoothing"? How about whether we included engravings in the "carve" phase?

[myk002]

Awesome work!! It would be better if there are rotation and flip options when applying blueprints.

This is now supported in quickfort. It will be available in the next DFHack release (or get it now from source : )