Skip to content
Branch: master
Find file Copy path
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
1002 lines (723 sloc) 34.2 KB

HOWTO - Hacking X4 Foundations

( Work in progress )

I'm still learning; apologies for any errors in this guide, please point out anything you see. At some point it will actually be a howto, currently it's notes links and blather.

Catch me (@h2odragon) on the x4 channels of the EGOSOFT Discord or on SnafuHall. Please feel free to PM me any links corrections suggestions additions criticisms or small wild animals.

Most of the code and data that governs the game is located in XML files. Mods can add to the game data, or even alter the stock data as its loaded.

The 'Nouns' in the game, the objects, ships, stations and parts of stations, NPCs, asteroids, and even map sectors are all defined as "macros" and "components", see the ships and stations section for more.

The 'Verbs', missions, ship behaviors, faction logic, NPC trader numbers and priorities, are all expressed in "ScriptXML". This is a domain specific language for scripting game actions. see the AIscripts and MD sections for more.

The user interface code, the map overlay, station builder, plot manager, etc; are all in Lua code. See ui and Lua section for more on that.

Many of the "I just want to change this one little thing" desires can be met with SirNukes' X4_Customizer. It can make it easier to change just one, or many little things; and give you a springboard for further creativity.

Debug Logging

To enable the games logging features, you'll need to specify some parameters at game launch. the "Properties" in steam has a "Set Launch Options" button that lets you enter them there, or you can create shortcuts that launch the game through steam with different options.

To invoke X4 with options from a desktop shortcut, set its target to

"C:\Program Files (x86)\Steam\Steam.exe" -applaunch 392160  -skipintro -debug all  -logfile debuglog.txt -scriptlogfiles

  • -applaunch 392160 - tells steam which game to launch. The rest of the options get sent on to that game, in this case, X4.

X4 Invocation Options:

  • -skipintro - just what you'd think

  • -debug all - turns on the debug log, and sets the log filter to "all". Other filter levels are error, general, scripts, scripts_verbose, economy_verbose, combat, and savegame. See the XSD entry for <debug_text>

  • -logfile debuglog.txt - specifies the name of the file the debug log is written to. This will be in the My Documents/Egosoft/X4/[digits]/ directory, beside your saves/.

  • -scriptlogfiles - turns on script logging, see the <debug_to_file> reference in common.xsd

  • -showfps - again, as you'd think

other options:

glancing through the binary there's also possibly -vsync, -clearstats, -disablecockpit, -disableallplayershiprendering, -clearstatsandachievements, -mouselooksteering

Unpacking game files

Euclid's forum thread Getting Started: Tools Scripting and Modding talks about unpacking the files. The more recent official tool or XERCES are options too. X4FProjector might also be of use.

.CAT and .DAT and mods

Packing your mod into this format is not required to use it in game, or publish for others to use. It can offer advantages and is required for distributing your mod on Steam Workshop, but can be handled later by the WorkshopTool.

Some Windows 7 "won't load" problems can be avoided by packing mod files into the .CAT format.

The readme file with the XRCatTool says (in part):

Notes for modders on catalog usage in X Rebirth, X Rebirth VR Edition and X4: Foundations:

At first, the game load catalogs of the form (## = 01..99). Loading stops at the first missing file.

In each enabled extension, the game then tries to load the following files:
- (starting with
- (### = current game version, e.g.
- (starting with
- (### = current game version, e.g.

The files in the ext_* catalogs are interpreted as part of the extension.
This is the recommended way of altering the game, it allows for a high degree of compatibility between mods.

The files in the subst_* catalogs are interpreted as relative to the game root (adding/replacing base game files).
This should only be used if there is no other way to adjust the game assets. Please make sure that players are aware of the
potential incompatibilities with other mods.

File Structure: What are we looking at?

However you get there, you should end up with a directory that looks like this:


also see the XR Wiki File Structure page

Why 01, 02, 03, etc? What's in there?

There's 8GB of data here, its quite a lot to go through. Three fourths of it is graphics and audio, and fortunately for us, everything is reasonably organized and the things we're likely to be interested in are informatively named.

I've unpacked into directories corresponding to the .DAT files, so my paths all begin with digits right now. This was a mistake, but as so often happens it can be an educational one. I will correct it later.

The game overlays the contents of each .DAT it loads and builds a view of the sum of all of them. In that view the 01/, 02/, etc directories don't exist, nor does the root directory of a mod being loaded.

The files in a mod get overlaid as their own DAT, even if they're not packed. My QuickBuilder mod consists of two files:


The content.xml tells the game about the mod; and libraries/wares.xml gets merged into the game's stock wares.xml file when the mod is loaded.

You can replace an existing ware completely this way, or you can use XMLpatch to modify existing values. Adding new content doesn't require <diff>.

The 01/assets/ and 05/assets/ also demonstrate the file overlaying. 05/assets/ redefines some ships previously defined in 01/assets/, example:

diff -u0 v2.50-unpack/01/assets/units/size_l/macros/ship_arg_l_destroyer_01_a_macro.xml v2.50-unpack/05/assets/units/size_l/macros/ship_arg_l_destroyer_01_a_macro.xml 
--- v2.50-unpack/01/assets/units/size_l/macros/ship_arg_l_destroyer_01_a_macro.xml	2019-05-28 19:19:27.975130500 -0500
+++ v2.50-unpack/05/assets/units/size_l/macros/ship_arg_l_destroyer_01_a_macro.xml	2019-05-28 19:38:01.144485000 -0500
@@ -2 +2 @@
-<!--Exported by: Michael ( at 21.11.2018_18-11-06-->
+<!--Exported by: nick ( at 10.05.2019_16-52-31-->
@@ -23 +23 @@
-      <people capacity="36" />
+      <people capacity="44" />
@@ -45,0 +46,3 @@
+      </connection>
+      <connection ref="con_shipstorage_xs_01">
+        <macro ref="shipstorage_gen_xs_01_macro" connection="object" />

To create a unified, unpacked file system that looks like what the game sees, I've used the command:

 for i in 01 02 03 04 05 06 07 08 09 ; do cp -Rvf $i/* ../v.250-unpack2/ ; done

and that results in the unpacked structure to use for reference in your own work. The only reason to unpack the .DAT files individually like I did is if you're a terminally curious person like me and want to see what gets overwritten.

[12:35 AM] h2odragon: @UniTrader  im trying to reference the vanilla assets by using <source geometry="assets\structures\connectionmodules\struct_arg_cross_01_data"/> in my component file. But from a mod that path wouldn't be correct?
[12:36 AM] UniTrader: it is correct 
12:36 AM] UniTrader: all file paths are relative to the X4 exe
[12:37 AM] UniTrader: this includes cat/dat archived files which are virtually extracted in the game root (01-09), in mod folder (ext_01) or again in the game root (subst_01)

The XSD files

First thing to look at is libraries/*.xsd, and scriptproperties.html. These files document the various program operations and variables you see used in the rest of the XML files. You want to keep them in mind.

I've got copies of UniTraders XSD files for some macro and component values as well as RFC 5261. These can be used with Code Complete and other editor tools.


The official name is 'ScriptXML' apparently.

The Mission Director guide section on script structure has a good explanation of how an MD script is built and is a clear example of how this is all wrapped in XML.

Gooey, chunky, XML, which when coupled with the "do what i meant" development ecosystems favored today can be off putting to those of us who are accustomed to text editors. Or code and data specifications having different syntaxes. But it works, and its truly not so bad. Better than working with a hex editor.

From CBJ on the forums:

Post by CBJ » Sat, 5. Aug 17, 18:41

Since all of it still applies, I will quote the answer I gave last time this discussion came up:

CBJ wrote: Some of the original reasons for choosing XML for the MD were:

  • Domain-specific structure (it's laid out in the form of the conditions and actions required for mission progress) and syntax (many of the conditions and actions are directly mapped to mission activities rather than just being generic programming terms)

  • Missions can be created with very limited programming knowledge (while it's possible to do programmer-y things like loops, it is also possible to create simple missions entirely without this, which helps with the learning curve)

  • Editing comes with built-in self-documentation as well as syntax auto-completion (most self-respecting XML editors understand schemas, and the schemas can also be used to provide the documentation)

  • Easy to parse and convert to internal format (it is loaded and parsed very simply using standard XML libraries, then converted into an internal form that is that is, like the raw syntax, closely tied to its purpose, rather than using a generic third party interpreter or JIT compiler)

Now, bearing in mind that we have been working with the MD, in one form or another, for nearly 10 years, the reason we continue to use it is very simple: we have it, it works, and it does what mission developers need it to. The AI scripting system shares some key elements, such as actions and conditions, with the MD, saving a lot of effort in maintenance and development, but is, again, tailored to the task in hand. It would be a very poor use of developer time to throw all that away in favour of something different, with all the issues and unknowns that would come with it, just because some people turn their noses up at it for being XML rather than a "proper" programming language.

Internationalization, or "why {1041,10146}"?

The files in t/ contain translations of strings used in the game. The files contain <page> sections, each of which is full of <t> tags. The two numbers in the identifier refer to <page id="X"> and <t id="X">.

For example, in aiscripts/ we see "{1041,10146}".

<param name="warebasket" required="true" default="if this.ship.job then this.ship.warebasket.list else null" type="list" text="{1041, 10146}" comment="Wares">
        <input_param name="type" value="'ware'"/>

The context makes it obvious, but: searching t/0001-l044.xml (which is the English language texts) we find:

<t id="10146">Wares</t>

The same method will be very useful for other less obvious strings.

True Names

Many people are bewildered at this point by the game's internal names for everything. Specifically, where to you find them and how do you know them.

You'll sort of sop things up organically eventually but thats slow and painful. Therefore I've put together lists of some of the more important concordances.

All the wares in libraries/wares.xml have anid attribute, which is the name by which you reference that ware in code. They have a tags attribute too, like container, mineable, modules, ship, equipment, inventory, research, or many others.

Ship Connections are important when dealing with ships in any detail.

Collecting Macros

Having a copy of "all the macros in one file" might be useful; here's a couple of bash commands to get that. In the unpacked game files directory:

find assets/ -name "*_macro.xml" -print > macrofiles
for i in `cat macrofiles` ; do echo -e "\n\n$i\n" ; cat $i ; done > allmacros.xml

The find command lists all the files under assets/ whose names contain "_macro.xml", and puts that list into a file named macrofiles. The for command reads the list of file names, then for each file prints out a header and the contents of the file; and all of that output is wrapped up into allmacros.txt.

find assets/ -name "*.xml" -and -not -name "*_macro.xml" -print > cptfiles

will get component file names.

What did you want to change?

Where you go from here depends on what you want to change. With the game data files on hand you can now begin downloading and unpacking (when necessary) other people's mods to look at. Do that. Do a lot of that.

Missions happen in md/, ship commands are governed by aiscripts/, wares are defined in librares/wares.xml. assets/ contains the definitions of all the ships, stations, rocks, etc.

filenames: all lowercase can avoid problems


The 'New Game' menu is made up of nodes found in libraries/gamestarts.xml. This has it's own XSD.

Creating your own gamestart involves adding new <gamestart> nodes by defining them in your mod's libraries/gamestarts.xml. See apricotstart for a template to begin from.

The md/ApricotStartSetup.xml file contains the mission director cues that do other things at the game's beginning. the id attribute of the <gamestart> node determines which _Init cue is called.

In the game files the setup is done in md/Setup_Gamestarts.xml, which contains more code for test starts than is accessible from the vanilla game. Here's a copy of the gamestarts.xml that goes with those. original link

The XR Custom gamestart howto might be useful too.

Laying out a ship in libraries/gamestarts.xml may require looking up the connection names for elements you wish to include (the path attribute). I've created table of ship connections for starts for that. It lists only the connections for engines, weapons, turrets, and shields, which you need in the <ship> section.



X Rebirth Mission Director Guide

MD scripts are used for other things, like the FlyBy Looting mod; which sets up a cue to fire every 10 seconds and check for loot around the ship.

mewosmith: all the globals from vanilla that are not debug


@mewosmith: you should use this as an example of why you need the xsd hooked up to code complete of some form

create_ship XmasTree example

Groups Are Magic

In the MD Guide, you'll find mention of the lists and table data types you'll see used in vanilla code. In the code you will also see "groups", which are most useful magic:

UniTrader: in short a group is a list with the following special properties inherent to it: UniTrader: => can only contain objects, nothing else UniTrader: => no duplicate entries UniTrader: => destroyed objects are auto-removed UniTrader: => can be used as event objects UniTrader: => if used in events it will work for all objects in the group, even if added after setting up the event


UniTrader: there is a $group.list, and you can in many cases treat a group like a list anyway. at least if you perform all your actions within a frame.


aiscripts/order.mining.routine.basic.xml is simple example of an default behavior aka infinite ship order. The non-infinite orders and subroutine scripts are different, more later. XXXX

Inside the <aiscript> tag is an <order> tag; (see the libraries/aiscripts.xsd for details of the attributes).

The <params> block defines parameters which may be presented to the user when they issue this order on one of their ships. Other more complicated orders will have <init>, <interrupts>, and may have <attention> nodes other than unknown.

<actions>; finally, some action! This is where your code begins to influence what a ship does. Execution arrives at this point

  • when the order is issued
  • after an interrupt has been handled
  • after the order script has completed or used <return />

Which means that a ship has to figure out what its doing after every gate transition, for example, fresh and without assuming anything about its current state.

A script can create and order with <create_order> (yet another example of reasonably named stuff). If the order is immediate, it will begin running right now, but your script will continue executing after the <create_order> node as well. cleanup, if needed, and exit (jump to finish) at that point and wait for the new order you created to complete, after which your default behavior will resume executing.

XXXX or possibly you could wait and periodically check this.ship.order?

Or you can run another aiscript/ with <run_script> (as is done in aiscripts/order.mining.routine.basic.xml) which behaves like a traditional subroutine.

A script referenced with <run_script> doesn't need the <order> tag in its structure.

On formations:

Re: Attention Levels
Sent: Mon, 5. Aug 19, 15:55 
From: j.harshaw 
Recipient: teleportationwars 


>teleportationwars wrote:Mon, 5. Aug 19, 14:28
> How can I get a fleet to travel drive in formation?

You can't yet, at least not using formation code. Formation code will
set positions for all formation wingmen relative to their commander,
so if their commander is moving rapidly using travel mode, they will
know that they have to keep up but they won't know to use travel mode
to do so. We have worked on this, however, and, in our current
internal build, they at least try to use travel mode to keep up, but
current public is missing code to have formation wingmen use travel
mode to maintain their positions. That code should be pushed out in a
future update, but i don't know when or which update exactly.

Closest you can probably do at the moment is to break formation and
approximate formation movement by periodically plotting positions
relative to a commander and having subordinates do move_to actions
with travel mode on or coordinating move_to actions between multiple
ships at the same time. That second in particular should work,
particularly for single, long move_to actions within a sector, but
they won't attempt to stay together while they're moving.

That said, i wouldn't be surprised if you or someone else figures out
a way to do it without it being officially supported. Part of the
beauty of having moddable systems.

>teleportationwars wrote:Mon, 5. Aug 19, 14:28
> How can I cross attention levels with a fleet held together by
> formation?

This is more problematic since movement in high attention is
fundamentally different to movement in low.

on <move_to finishonappoach="true">: the XSD explains this as "Whether the object should stop if the movement is approaching to finish (defaults to false)" which still leaves some room from interpretation.

UniTrader: @h2odragon finishonapproah means "im close enough that i coult exit this command and start the next one"

UniTrader: but its not fixed to zones or distance. or maybe loosely to the latter. for slow moving ships it was maybe a few 10 to 100 meters, for boosting ones about 1-2 km when this command returned with that (about 10 to 15s travel time remaining i think)

UniTrader: dont think its exact. i would say the condition is checked every few seconds only

UniTrader: (note: knowledge from experimenting with it in XR)

h2odragon: "about 10s of travel time left under current conditions" ? sound good?

UniTrader: its more a guesstimate, but yeah

UniTrader: its certainly enough to start the next movement step, depending on wheter you set abortpath to true or false the change will be more or less aprupt

UniTrader: (i recommend the latter if you are close to the last waypoint and want fluent movement, except in specific circumstates)

ships and stations

The assets/ folder contains the definitions of in game items from stations, asteroids, ships, all the way down to the models for crystals and bullets. *_macro.xml files (usually in a macros/ subdirectory) reference the component files to bind things like the base damage of a bullet to the graphics effect for that bullet.

BrummBear's Building your own ship for dummies

How the ecell plant is defined in vanilla

  • wares.xml - defines the blueprint ware

    • <ware id="module_gen_prod_energycells_01"
    • includes <component ref="prod_gen_energycells_macro" - aka 'the macro'
  • index/macros.xml - maps macro name to macro node

    <entry name="prod_gen_energycells_macro" 
 	   value="assets\structures\production\macros\prod_gen_energycells_macro" />
  • index/components.xml - maps component name to model data
	<entry name="prod_gen_energycells" value="assets\structures\production\prod_gen_energycells"/>
  • assets/structures/production/macros/prod_gen_energycells_macro.xml - this station's macro definition
<macro name="prod_gen_energycells_macro" class="production">
    <component ref="prod_gen_energycells" />
  • assets/structures/production/prod_gen_energycells.xml - defines connections and references model
<component name="prod_gen_energycells" class="production">
<source geometry="assets\structures\production\prod_gen_energycells_data"/>
  • assets/structures/production/prod_gen_energycells_data/ - actual compiled model data
    • *.xmf including assets_structures_production_prod_gen_energycells-[collision,lod0].xmf

positions and coordinates

from libraries/macro.xml:

<macro name="boundaryexamplezone" class="zone">
        <!-- Position and rotation define the local coordinate system in which the box goes from (0,0,0) to the vectors defined by size. -->
        <boundary class="box">
          <position x="-2000" y="-1000" z="-2000" />
          <rotation yaw="45" pitch="45" roll="45" />
          <size x="4000" y="3000" z="4000" />
        <!-- Pretty self-explanatory... -->
        <boundary class="sphere">
          <position x="0" y="0" z="5000" />
          <size r="100" />
        <!-- Again, position and rotation define the local coordinate system. The linear size is the height of the cylinder, going in the local y-direction. -->
        <boundary class="cylinder">
          <position x="1000" y="0" z="-1000" />
          <rotation yaw="0" pitch="-90" roll="0" />
          <size r="1500" linear="15000" />
        <!-- The 4 positions define the 4 points of the spline in the order Start, Control1, Control2, End. Size gives the radius of the tube. -->
        <!-- The class splits the tube into a number (given by the 'parts' attribute) of cylinders. -->
        <boundary class="splinetube" parts="20">
          <splineposition x="0" y="-6000" z="0" />
          <splineposition x="2000" y="-2000" z="-500" />
          <splineposition x="2000" y="2000" z="500" />
          <splineposition x="0" y="6000" z="0" />
          <size r="1000" />
    <component ref="testzone" />


weapons and turrets are defined in assets/props/WeaponSystems/*

bullets are defined in assets/fx/weaponFx/.

@mewosmith tip: Values in weapon macros (damage etc) using min and max attributes for randomness will also require a value attribute. This is apparently the only place value is required and not optional.

 mewosmith: @DocAce   bullet_xen_s_laser_01_mk1_macro has 
 <damage min="16" max="24" value="18" repair="0" /> 
 It would be very cool to balance weapons with min and max. How can I make missile launchers/turrets burstfire? Something like 1 every 0.5s for 10s then 30s on cooldown.
 DocAce: Huh, it does do something. The damage value is interpolated between the max and min values using a decrease time or rate you can specify. (If you set one the other is automatically calculated.) I think it's ineffective in this example macro because it's not set.
Let me just give you a full damage node example, all values at their defaults:
<damage min="0" max="0" value="0" hull="0" shield="0" noshield="0" delay="0" time="0" rate="0" />
Bullets then add some properties of their own:
<damage repair="false">
  <multiplier mining="1" />
 DocAce: If I understand the code correctly it's decreasing the damage based on the time after bullet launch. You could effectively do range scaling with this.


enenra's universe creation page

DeadAirGate Overhaul

DeadAir's Sector map with IDs - has ID numbers of vannila clusters plus DAG's new sectors.


(2019-07-05) mewosmith's effectnames.xml - Test harness and notes on included effects


Appearance is governed in libraries/character_macros.xml Helmets can be removed there, bodies and types changed.

Character names are found in the text pages, and the random name generation (using the <identification name="@random" notation unique to that file) has been explained:

8:17 AM] DocAce: The @random name tells the code to randomly generate one for the character, using the race's text name page. That page also specifies the rules for how names are generated, new options should be reasonably easy to mod in
[8:18 AM] DocAce: Pages 20301 and following are what you should be
looking at
8:20 AM] UniTrader: First entry is always the formatting. Then come
the amount of entries for each name part and seperated by hundreds are
the parts themselves (iirc; could be mistaken)
8:23 AM] DocAce: Hm, you might actually find it difficult to add new names after all, there doesn't seem to be much space left in some of the pages
[8:24 AM] DocAce: Ah, I remember now. I added a few names myself for
X4 and ran into that limitation. Had to cut a few names to make it fit
into 100

ui / Lua

How do we run Lua ; what's the Gnull thing about?

The intended method of adding lua files is to place a ui.xml file in
the extension's primary folder, which in turn specifies the lua files
to load into the game.

As of X4 2.5, lua files loaded this way are provided some basic X4
functions (eg. DebugError), but their globals table is not

Without this table, the lua code cannot access the various UI
functions exported by other X4 lua files, lacks FFI support, and lacks
a way to communicate with the mission director.

Even basic lua functions are unavailable.

A workaround is to load in custom lua files alongside the egosoft lua.
This is done by editing one of a handful of ui.xml files in the
ui/addons folders, adding the path to the custom lua file.

These ui.xml files cannot be diff patched.

The lua file must be given an xpl extension, and this xpl and the
ui.xml must be packed in a "subst" cat/dat.

Since there are a limited number of such ui.xml files, there is a high
likelyhood of conflicts in mods importing lua files this way.

Prior work

An initial workaround was provided by morbideth

However, this was presented alongside a much more complicated
right-click-menu mod, leading to confusion among modders as to how to
use the workaround, how reliable it is, and if they had permission to
use it.

This new api has a new, somewhat simpler implementation, and is
available under the MIT license.

Nodus' Lua lists:

Windows 7 issues

I personally run the game on Windows 7 and develop mods to run with it. Quoth the wiki "Many mods will not work with Win7". Personal observations show this to be true. Specific causes of Win7 issues I've observed and worked around include:

  • XMLPatch diffs can replace entire nodes, but not attribute values.

  • Files in index/ can't be touched without the mod is packed into .CAT form

  • I couldn't do anything (XMLPatch or add nodes) to lens_effects.xml except replace it entire via

  • A mod that won't work on Win7 makes the game not reach the main menu, usually. Sometimes the point at which it hangs and the memory footprint it has at that point varies on successive runs with identical inputs. This means something, perhaps someone can tell me what.

Making Diffs

it may be useful to have diffs of the game files when new versions are released. This is how i did it for 2.50 to 2.60:

diff -ru0X asset-xpat  v2.50-unpack/assets/ v2.60-unpack/assets/ > assets-25v26.diff
for i in aiscripts md libraries ; do  diff -ru0 v2.50-unpack/$i v2.60-unpack/$i > $i-25v26.diff ; done
diff -ru0 v2.50-unpack/libraries/wares.xml v2.60-unpack/libraries/wares.xml > wares.xml-25v26.diff 

the file asset-xpat contains:

You can’t perform that action at this time.