Factorio-A11y (pronounced "factorio ally") is an accessibility mod and associated voice control grammar for Factorio, which aims to make it possible to play the game with a little as possible input from mouse and keyboard while preserving the spirit of the game.
Paired with voice freely available voice recognition software, you can say:
refuel everythingto put the best fuel you have into every furnace, car, burner inserter/mining drill in your reach
run there, grab stone furnace, click, refuel itto run where the cursor is, get a stone furnace from your inventory, build it, and load it up with the best fuel you have.
mine here repple five, craft ten wooden chestto mine the 5 closest trees and craft ten wooden chests
A11y aims to be compatible with other mods and to preserve existing gameplay mechanics; e.g. although it adds a way to move the character via the mouse cursor, rather than having the character instantly teleport from point A to point B, the character still needs to walk there along a valid path and brick/concrete still provide relevant speed boosts.
Currently the mod is done enough to use for the early (pre combat) game, with new features being added regularly. Star this repository if you're interested to help me prioritize my projects.
Voice Control Software (required)
Factorio-A11y expects you to use voice control software; as of early 2019, the main options available are:
- Dragonfly (Windows, some Linux support too) - supports Windows Speech Recognition, Dragon Naturally Speaking 13 or higher, or (experimentally) the open source Sphinx speech recognition system.
- Vocola (Windows) - requires Dragon Naturally Speaking 13 or higher.
- Talon (MacOS) - has a built in speech recognition engine, but can make use of Dragon Naturally Speaking 6.
You only need to set up and install one of these.
Mouse alternatives and replacements (optional)
If you're interested in this mod for serious use, you might also want to check out:
- Vertical mice, graphics tablet pens and trackballs might be less painful to your hands.
- JoyToKey or Gopher360 can remap controllers/joysticks to send keypresses and/or mouse movements/clicks
- eViacam (free, open source, Windows/Linux) can use a standard webcam to control the cursor by tracking your face so that the cursor moves when your head moves
- Precision-Gaze (free, open source, Windows) is better, but requires you to buy eye or head tracking hardware (usually available for < 200 USD).
- Talon (free, MacOS) requires you to buy eye tracking hardware (usually available for < 200 USD), but is probably the most advanced option available (also supports making mouth noises to click).
If you can use a mouse fine and just want to use voice control for convenience, then you can ignore these. (Although it is very cool to control a game using only your eyes!)
Ok I'm in, how do I make it work!?
- Install the mod into Factorio: follow the setup instructions in
- Install a Factorio voice grammar (a set of voice commands for some software):
- If you use Dragonfly, follow the installation instructions in
- If you use Vocola or Talon, you'll have to first port the Dragonfly grammar over. Send a PR when you're done!
- Start controlling Factorio by voice!
- Start by saying
data reloadto get the voice grammar to know about all the Factorio items, recipes, etc loaded into the game.
A voice grammar is a collection of a set of voice commands and associated logic.
Factorio-A11y assumes you already have a grammar for regular clicking, right clicking, hitting individual keyboard keys (to open/close inventory/map/etc), etc.
With that out of the way, Factorio-A11y provides the following commands:
The voice grammar loads data from Factorio, but you need to ask it to do so:
data reloadwill force Factorio to dump data which the voice grammar will then read
data list itemswill list out names of known items (things you can
data list recipeswill list out names of known recipes (things you can
Basically you should say
data reload anytime after unloading & reloading your voice grammar, after updating Factorio, or after changing your Factorio mods.
explain itwill print out the name of what you're holding or hovering over, which is very useful in combination with other commands.
run therewill run your character to where your cursor is on the screen.
run <direction>will run your character in the given direction indefinitely (useful for exploring)
directionis any 1-2 of
stopwill stop your character from running in a direction indefinitely.
mine orewill mine the closest resource in reach which the player can mine (ore, tree or rock)
mine housewill mine the closest building in reach
mine itwill mine the item being hovered as long as it is in reach (building, resource, vehicle, etc)
mine tilewill mine the tile being stood on (bricks, concrete, etc)
craft <count> itwill craft the item being held or hovered over
<count>is the number of times you want to craft the recipe which creates the hovered item
craft <count> <recipe>will craft the item which
<count>is the number of times you want to craft the recipe
<recipe>is the name of an item you can craft
- For example:
craft 2 transport beltwill craft two regular conveyor belts.
pastewill Shift + Left/Right Click respectively (and hence work as per Manipulating Entities).
<rotate_count>is an optional number from 1-4) will rotate/reverse-rotate an entity 1-4 times.
- For example,
red 2will reverse the facing of a building.
- For example,
splitwill Ctrl + Left/Right Click respectively (and hence work as per Manipulating Entities).
grab <item>will transfer that item from your inventory to your cursor (so you can build it or put it into an assembly machine, etc)
<item>is the name of an item you have in your inventory
- For example:
grab iron platewill put a stack of iron plates into your hand.
refuel herewill refuel the closest refuelable entity
refuel itwill refuel the entity being hovered over (building, vehicle, etc)
refuel everythingwill refuel
- Refueling works with buildings (burner miners, burner inserters, stone furnaces, etc) and vehicles (cars, tanks, trains).
- The highest energy value fuel is used first (coal before wood, for example).
- The correct fuel for each entity is calculated dynamically, not hardcoded, so this should work with mods.
- Can we see what someone is hovering the UI? Would be useful to "what is" a hovered item in inventory.
QoL and papercuts
- Make voice grammar detect Factorio script output location? Or at least read it from config file.
- Voice grammar should save and load last data loaded from Factorio and restore it upon grammar reload
- When you can't craft something because of missing resources, print out how many of what is missing
- Restore the thing in hand after a "run there"
- When mining nearest resource, prefer mining things that collide with the player (e.g. trees & rocks) for convenience in clearing a path.
- Have a way to grab all of an item in range quickly (both from floor and from inventories of items)
- Have a way to mine everything in range quickly (for clearing trees)
- Have a way to lay belt (assuming belt is in hand) from first to last click (assuming it's in a row). Maybe show a cross of visual lines as a guide to help line up tiles? Will probably need to check that each tile can be built before starting, and if we fail to build anything then stop building and print an error.
- Allow aliasing virtual items when crafting or grabbing? E.g. "craft/grab electric"
- oil is not mineable, so should be filtered out from the mining UI. Is there a generic API for detecting non-mineable objects? Looks like the player prototype has a mining_categories field: https://wiki.factorio.com/Prototype/Character and it defaults to
basic-solidaccording to a data raw dump I found.
- Mining resources and buildings should take some time - implement the mining hardness formula for this based on the FF post.
Voice control grammar notes
It's important to have a uniform interface to perform actions on game entities, so here are thoughts on voice grammar design to inform the API of A11y:
- Most actions should be of the form
- Actions are things like "mine", "craft", "grab", "run", etc
- Certain actions might have numbers after them - e.g. "craft 15"
- Targets should link to game objects/entities. To support both eye/head-tracking and voice-only
playstyles, we need to support several targeting mechanisms:
it- whatever cursor hovers over (or chose using a selection tool)
here- target closest entity eligible for action
all- target all entities eligible for action
<item prototype name>- target the entities whose prototype is named this. E.g.
grab copper plateor
mine iron ore
<item prototype group>- some items are part of the same named "group" (
game.player.selected.prototype.subgroup.name), like all trees are
grid <coordinate>- to enable true voice-only play, we need to be able to move and interact via something like a grid system, or naming tiles via tiny UI.
- should mining resources use different keyword than deconstructing buildings? They're the same as far as the game is concerned, but it might be helpful to allow mining items
- there needs to be a name for mining tiles, separate from mining resources
- we have an indicator for the closest resource, and closest building still needs implementing.. but what about tiles? Since the only interesting thing to do with a tile is mine it, maybe this should only be drawn when there's a tile in mining range?
- should the grid system use rows & columns like B3 or should each tile just have its own number? how do these scale for longer reach or interacting outside reach? Due to latency we want to avoid a dragon-mousegrid-like system.
- sometimes you're okay with your character moving to fulfil the command, sometimes not. Should there be
a modifier suffix, like
<action> <target> [<modifier>], where you can say things like
movingto allow your character to move?
-- convert a table to a string -- see https://github.com/pkulchenko/serpent game.player.print(serpent.block(p)) -- alternative game.player.print(inspect(p)) -- drop a traceback debug.traceback() -- writes to the game's log log() -- writes to stdout (run the game from a terminal) print()
Thank you to: