Knowledge Nuggets

This page contains snippets of useful knowledge and tips related to Ballistica development that are not big enough to warrant their own page. Feel free to add your own here or upvote existing ones by incrementing the 👍 count next to them (just once per tip please). I will use upvote counts to help organize the tips by usefulness.

Nugget List:

• Mypy: Getting a Variable's Type
• Debug-Only Python Code
• Beware The 'Any' Type
• Python Type-Checking Boilerplate
• Data Classes are Handy
• Enums are Handy
• Makefile Autocompletion
• Python Cache Files and Release Builds

Wish List:

Is there something you'd like to see here? Add it to this list or increment the 👍 count on an existing one.

• How do I create my own Spaz type? 👍 0

---

Mypy: Getting a Variable's Type

When using Mypy to perform static type-checking on Python code, one of the most basic and important things to be able to do is determine the static type of a variable as Mypy sees it.

Say we have the following code:

value = some_mysterious_function()

How do we know what type Mypy thinks 'value' is? We could guess this ourself by looking up the definition of some_mysterious_function(), but it is often better to be sure and just ask Mypy directly. We can do that by adding the following special fake function call to our code which is understood by Mypy:

value = some_mysterious_function()
reveal_type(value)

Now we can run make mypy in a terminal from the project root, and we should see output such as this:

Running Mypy (incremental)...
path/to/this/python_script.py:45: note: Revealed type is 'builtins.int'
Mypy: fail.
make: *** [mypy] Error 255

Ok, Mypy says 'value' is an int. Good to know! So if we were to add a statement such as value += 'foo', then Mypy would kindly inform us that ints and strings cannot be added. Without type checking, an error such as that might go unnoticed until the next time the code gets run, and who knows when that might be. Hooray for static type checking!

Be sure to remove the reveal_type() line when you are done with it, otherwise the code will error at runtime due to that not being an actual Python function.

👍0

Debug-Only Python Code

With the Ballistica project it is now easy to switch between debug and release builds of the game. (make prefab-debug vs make prefab-release). Debug builds perform much more error checking than release builds; this makes them very useful during development, with the tradeoff that they can be significantly slower and less efficient than release builds.

You can take advantage of having two build types by adding extra safety checks that run only in debug builds. There are a few ways to do this.

The first is the Python's 'assert' statement. These will be evaluated in debug builds and completely stripped out of release builds, so think of them as 'free'. They are a great way to make sure things are as you expect them to be.

my_thingie = make_a_thingie()
assert my_thingie is not None. # <-- Throws an AssertionError if not True.

As a nice side-effect, assert statements are also quite useful for Mypy type-checking. (see Mypy: Getting a Variable's Type)

# Dynamically evaluate a string to create an object (in this case an int).
val = eval('123')

# If we run 'make mypy' from the project root, Mypy will tell
# us that val is 'Any' here because eval() can return anything.
reveal_type(val)  

# We expect val to be an int; let's make sure that's the case.
assert isinstance(val, int). 

# Because of the above assert statement, Mypy knows val must be an int
# at this point, so 'make mypy' will show 'builtins.int' here.
reveal_type(val). 

Another way to generate debug-only code is with Python's special __debug__ variable. Code such as this will run in debug builds of the game but will be completely stripped out of release builds, again making it 'free'.

if __debug__:
    for node in all_my_nodes:
        do_some_expensive_sanity_checks(node)

With any code such as this, it is extremely important that no actual logic be affected as a side-effect of these expressions since they will not be run at all in release builds.

👍0

Beware The 'Any' Type

Once you have familiarized yourself with checking variable types, the next important thing to remember with type checking to make sure things actually have types. There is a special 'Any' type which Mypy will give variables when it doesn't know what type they are. Mypy will never generate errors for 'Any' variables since it doesn't know what is or is not valid for those objects, and thus that code does not benefit at all from type checking. For type checking to work well, it is important to watch out for 'Any' values and inform Mypy what types they are supposed to be.

One common example is objects contained in a dict:

func handle_thingies(thingies: Dict):
   # This argument declaration doesn't specify the types the 'thingies' dict
   # contains, so Mypy will consider its members to be 'Any'.

   thingie1 = thingies['first']
   reveal_type(thingie1)  # <-- Mypy will tell us thingie1 is 'Any' here.

   # Now if we do something illegal with thingie1, Mypy will not warn us because
   # it doesn't know what type thingie1 is, so it has no idea what is ok to do.
   # We won't know about this problem until we hit it at runtime, which is what
   # type-checking is supposed to save us from.
   thingie1.method_that_doesnt_exist()  # <-- 'this is fine!' says Mypy.

   # We can improve the situation by telling Mypy what type thingie1 is. One
   # way to do this is with the 'assert' statement, (as discussed in a previous
   # example). This has the added benefit of double-checking at runtime in
   # debug builds that the object is actually the type we think it is.
   assert isinstance(thingie1, Thingie)
   reveal_type(thingie1)  # <-- Now Mypy knows thingie1 is a Thingie.
   thingie1.method_that_doesnt_exist()  # <-- Now Mypy will correctly error here.

   # We could also give our variable a type when we declare it.
   # Mypy considers everything in thingies to be 'Any' so it assumes this
   # assignment is valid.
   thingie2: Thingie = thingies['second']
   reveal_type(thingie2)  # <-- Mypy says this is a Thingie.

   # Note that it could still be good to add an assert() statement as a runtime
   # sanity check in this case, since Mypy thinking something is a certain type
   # does not necessarily mean that's what it will be at runtime. assert() statements
   # can help make sure the type-checking world and runtime world are actually in sync.
   assert isinstance(thingie2, Thingie)  # Just checking!

   # One more tool for wrangling types is the 'cast' function in the typing module.
   # At runtime this simply returns the object passed to it, but in type checking
   # it changes the type an object is considered to be.
   from typing import cast, List
   extra_thingies = thingies['extras']
   reveal_type(extra_thingies)  # <-- Once again Mypy thinks this is 'Any'.
   extra_thingies2 = cast(List[Thingie], extra_thingies)
   reveal_type(extra_thingies2)  # <-- Now Mypy says this is a List[Thingie].

   # Note that it seems a new variable name must be used for the result of the cast;
   # if extra_thingies is reassigned to instead, it seems to remain as Any.

So the basic morale of this story is: hunt down and destroy all occurrences of 'Any' in your code and you will get the maximum benefit from type checking.

👍0

Python Type-Checking Boilerplate

You may notice certain things at the top of most Ballistica Python modules; namely this code:

from __future__ import annotations

from typing import TYPE_CHECKING

if TYPE_CHECKING:
    import xxx

The following is an explanation of why that code exists:

Ballistica uses Mypy for static typing. Static typing lets us catch a large number of issues in our code without having to run it. But for this to work, we need to add type annotations to our functions and variables.

# So instead of this:
def get_next_number(input):
    return input + 1

# We need this:
def get_next_number(input: int) -> int:
    return input + 1

Pretty simple so far. However, sometimes we can't annotate things because the names we want to use don't exist yet. Consider the following where we want to create a 'my_thingie' variable that can either point to None or a Thingie instance:

from typing import Optional

my_thingie: Optional[Thingie] = None

class Thingie:
    pass

If we run this code, it will fail, saying 'Thingie' is not defined on the line where we create my_thingie. This is true; it won't exist until Python executes the code 2 lines down from it which defines class Thingie. And yes, in this case we could simply move the class definition above the variable definition, but ignore that for the sake of this exercise.

Python type annotations work around problems like this by allowing forward declarations using strings:

from typing import Optional

my_thingie: "Optional[Thingie]" = None

class Thingie:
    pass

This code will work, but it makes the annotation look ugly. Luckily, because Ballistica uses Python 3.7+, we can make this cleaner by using Postponed Evaluation of Annotations. This essentially turns all annotations into strings under-the-hood, so we can write more natural looking code but still forward-declare things. This also makes things more efficient since Python doesn't have to evaluate complex annotation code such as List[Dict[str, Optional[Thingie]] at runtime; it simply sees that whole thing as a string. This behavior will be default in a future version of Python, but for now we have to use a __future__ import to get it.

So now our code looks like:

from __future__ import annotations

from typing import Optional

my_thingie: Optional[Thingie] = None

class Thingie:
    pass

This code runs and isn't too ugly (aside from the __future__ bit which we will eventually be able to get rid of). But there is one final issue. Running make check on this code will give us an error, with Pylint telling us that the imported 'Optional' is unused here. This is technically true because my_thingie's annotation is really just a string now; no actual runtime code is using Optional. However it is needed by the type-checker; if we remove that line and run make check again, then Mypy type checking will fail saying Optional is not defined. So how do we make everyone happy here?

The answer is to define such things only for the type checker. We can do that using a special TYPE_CHECKING constant provided by the typing module. So the final form of our code is this:

from __future__ import annotations

from typing import TYPE_CHECKING

if TYPE_CHECKING:
   from typing import Optional

my_thingie: Optional[Thingie] = None

class Thingie:
    pass

So now everyone is happy; we're not being wasteful at runtime by importing things we don't use, but the type checker is still able to find everything it needs. This leaves us with a few lines of boilerplate code per module, but it gives us a good type-checking foundation. With this setup we can isolate imports and other code that is only used for type-checking, keep forward-declarations clean and readable, and minimize the impact of type-checking on runtime performance.

👍0

Data Classes are Handy

Python 3.7 introduced data classes, so we now have access to these in Ballistica. These work very well with our type-checking setup, allowing us to easily define little bits of data that get passed around. We could also write full classes for these purposes, but that is often overkill and would require extra boilerplate code.

Here is how we might have done some things before, using simple dicts and values:

# Hmm; let's keep track of some spawn points for our mini-game or whatnot...
spawn_points = []
spawn_points.append({'position':(0.0, 0.0, 0.0),
                     'team':0})
spawn_points.append({'position':(0.0, 0.0, 0.0),
                     'team':1,
                     'color':(1.0, 0.0, 0.0)})

# That works, but it can be easy to introduce subtle errors that
# won't be noticed until runtime:

# Oops; wrong name for the dict entry.
spawn_something_at_position(spawn_points[0]['pos'])

# Oops; our first spawn point doesn't contain 'color'; we'd need to
# check for that case and substitute a fallback value here.
flash_at_position(spawn_points[0]['position'],
                  color=spawn_points[0]['color'])

Using data classes, we can now write this code in a much safer manner:

from dataclasses import dataclass
from typing import Tuple, List

@dataclass
class SpawnPoint:
    position: Tuple[float, float, float]
    team: int
    color: Tuple[float, float, float] = (1.0, 0.0, 0.0)

spawn_points: List[SpawnPoint] = []
spawn_points.append(SpawnPoint(position=(0.0, 0.0, 0.0), team=0)

# Now running `make check` will catch all sorts of errors for us without
# having to run the code:

# Error: required 'team' arg not provided.
spawn_points.append(SpawnPoint(position=(0.0, 0.0, 0.0))

# Error: SpawnPoint has no 'pos' attr.
spawn_something_at_position(spawn_points[0].pos)

# And this WILL work now because SpawnPoint gets a 'color' value
# by default even if it wasn't passed in the constructor.
flash_at_position(spawn_points[0].position,
                  color=spawn_points[0].color)

Hopefully this shows how much safer code is when defined in this way. It makes it possible to add, remove, or rename attributes from SpawnPoint later and instantly see exactly which places in the code need to be changed, and it also just feels cleaner and more readable without so many brackets and quoted string values everywhere.

👍0

Enums are Handy

Python 3.4 introduced enumerations, so we now have access to these in Ballistica. These can work well when paired with our type-checking setup, allowing us to write safer code.

Consider the following situation: we might want to write a mini-game that involves spawning bombs at particular places in a map. In the old system we may have written something like this:

def spawn_bomb(where):
    if where == 'top_left':
        pos = (-10, 0, 10)
    elif where == 'top_right':
        pos = (10, 0, 10)
    elif where == 'bottom_left':
        pos = (-10, 0, -10)
    elif where == 'bottom_right':
        pos = (-10, 0, 10)
    else:
        raise ValueError(f'Invalid where: {where}')
    do_spawn_bomb_at_position(pos)

# This first line will work, but the second will error at runtime.
spawn_bomb('top_left')
spawn_bomb('center_left')  # OOPS; not a valid value.

That works, but if we want to modify our mini-game with a different set of 'where' options then we have to carefully go through any code that calls spawn_bomb() to make sure none are passing no-longer-valid string values. This is exactly the sort of thing that type-checking can do for us.

Rewriting this to be type-safe using enums looks like:

from enum import Enum

class SpawnPos(Enum):
    TOP_LEFT = 'tl'
    TOP_RIGHT = 'tr'
    BOTTOM_LEFT = 'bl'
    BOTTOM_RIGHT = 'br'

def spawn_bomb(where: SpawnPos) -> None:
    if where is SpawnPos.TOP_LEFT:
        pos = (-10, 0, 10)
    elif where is SpawnPos.TOP_RIGHT:
        pos = (10, 0, 10)
    elif where is SpawnPos.BOTTOM_LEFT:
        pos = (-10, 0, -10)
    elif where is SpawnPos.BOTTOM_RIGHT:
        pos = (-10, 0, 10)
    else:
        raise ValueError(f'Invalid where: {where}')
    do_spawn_bomb_at_position(pos)

# This time around, the error on the second line gets picked up
# by type-checking; no need to run the code.
spawn_bomb(SpawnPos.TOP_LEFT)
spawn_bomb(SpawnPos.CENTER_LEFT)  # OOPS; not a valid value.

Now make check should ensure that all calls to spawn_bomb() are getting passed valid SpawnPos objects, so if we add or remove values from SpawnPos we can instantly see which code needs to be updated.

The values associated with enums can have basic types such as int, str, etc. and can be useful when storing the enum as JSON, passing it over the network, etc.

enumval = SpawnPos.TOP_LEFT  # This is an enum object.
rawval = spawnpos.value  # This gives us the string value 'tl'.
enumval2 = SpawnPos(rawval)  # Recreate SpawnPos.TOP_LEFT from 'tl'.

See the enumerations docs for more tricks.

👍0

Makefile Autocompletion

Ballistica's main Makefile is set up to list info about its available targets when you simply type make or make help. However, it can also be handy to set up your shell to allow autocompleting target names. If you are using zsh (the default on Mac as of 10.15 Catalina), you can add this to your .zshrc file to enable it:

autoload -U compinit
compinit

Once you restart your shell, you should be able to type something like make prefab- and hit tab to see a list of all matching targets.

ericf@MacBook-Fro ballistica % make prefab-
prefab-debug                  prefab-mac-release
prefab-debug-build            prefab-mac-release-build
prefab-linux-debug            prefab-release
prefab-linux-debug-build      prefab-release-build
prefab-linux-release          prefab-windows-debug
prefab-linux-release-build    prefab-windows-debug-build
prefab-mac-debug              prefab-windows-release
prefab-mac-debug-build        prefab-windows-release-build

The same functionality should be possible with other shells such as bash, but I will leave that as an exercise for the user (feel free to expand this tip).

👍0

Python Cache Files and Release Builds

TLDR: In some specific cases you may need to blow away .pyc files if you are editing .py files.

When Python imports a module, it attempts to first create an intermediate 'compiled' .pyc file from the raw .py script file. This compiled version can then be reused to import the module more efficiently the next time it is used. In old 2.x versions of Python these .pyc files were created alongside the .py files, and in modern 3.x versions they are placed in a __pycache__ directory alongside the .py files.

Traditionally, Python either recreates a .pyc file or ignores it whenever the timestamp on the corresponding .py file differs from it. This generally does the right thing as long as Python has write access to the .pyc file, keeping the .pyc up to date whenever the .py changes. However, in cases such as game distributions, Python might not have write access to the .pyc files, and bundled .pyc files might end up with timestamps that differ from their .py files as a result of the install process. The combination of these can lead to inefficiencies as Python repeatedly tries (and fails) to recreate these .pyc files. See PEP 552 for more details.

Ballistica now avoids this problem by creating optimized 'unchecked hash' .pyc files as part of its build process. 'Release' builds of the game will look for .opt-1.pyc files for any loaded module, and if one is found, it will always be used, even if the corresponding .py file is subsequently changed. This means these .opt-1.pyc files must be explicitly updated instead of relying on Python to do so.

You don't need to worry about this if you are using Ballistica Makefile targets such as make prefab-release; .opt-1.pyc files will be automatically regenerated as part of these builds. But if you are hacking on scripts directly in an installed copy of the game, be aware that you will need to blow away the script's existing .opt-1.pyc file or your edits will never be seen by the game. (though you should only need to do this once; any new .opt-1.pyc files written by the game should be timestamp based)

Note that this only applies to the release build of the game. Debug builds still use plain old timestamp-based .pyc files and will continue to exhibit the classic behavior.

👍0