Skip to content

Census API Primer

Jump to bottom
Leonhard S edited this page Feb 22, 2024 · 11 revisions

The following is a rewritten and updated version of the DBG documentation. It currently only covers the REST API, but strives to provide more detailed examples and tips for new developers.

Please do get in touch via the repository issues for corrections, feedback or additional example queries or joins.

REST API Endpoint

The base URL for all queries to the Census REST API is https://census.daybreakgames.com/. Visiting the bare URL will display an introductory page similar to this one.

The following structure is used to access game data via this endpoint:

https://census.daybreakgames.com/[<service_id>/][<format>/]<verb>/<namespace>/<collection>/<query>

Service ID

Service IDs are used by the API endpoint to identify the services accessing it.

If no service ID is specified, the default service ID, s:example will be used instead. This service ID is available to anyone and provides full access to the API but is limited to only ~10 requests per minute.

To lift this limitation, you need to apply for your own, private service ID using this form. A few hours after submission, you will receive an email with your private service ID.

Important: Your service ID is private. It is used to identify your application, so be sure to keep it in a safe place - you do not want to accidentally push it onto a GitHub repository.

Format

This primer will not use the format part of the query syntax as the JSON default is more efficient and easier to navigate, especially thanks to Python's built-in json module. To use XML output format, insert xml/ into the query, just before the query verb.

Verb

The query verb decides the type of query to perform. Valid values are get, which performs regular queries, and count, which only returns the number of results matching the query string.

Namespace

The request's namespace (aka. "game") tells the server which database to forward the query to. The following namespaces are currently published:

Namespace Game version
dcuo DC Universe Online (PC and PS3)
eq2 EverQuest II
global Server status information, any game
ps2 PlanetSide 2 (PC)
ps2ps4us PlanetSide 2 (PS4) - US Server
ps2ps4eu PlanetSide 2 (PS4) - EU Server

Note: The deprecated legacy namespaces like ps2:v1 seem to have been taken offline as of Jan 2019.

Collections

A collection is analogous to a table in the server's database. They are generally namespace-specific, although the PlanetSide 2 namespaces ps2, ps2ps4eu and ps2ps4us generally share the same object model (there might still be differences though as updates are not rolled out at the same time across platforms).

Listing Available Collections

When accessing a namespace without specifying a collection, the server will return information about the collections available for that namespace:

Namespace Count Link
dcuo 30 https://census.daybreakgames.com/get/dcuo/
eq2 33 https://census.daybreakgames.com/get/ep2/
ps2 111 https://census.daybreakgames.com/get/ps2/
ps2ps4us 111 https://census.daybreakgames.com/get/ps2ps2eu/
ps2ps4eu 111 https://census.daybreakgames.com/get/ps2ps2us/

These URLs return JSON documents containing lists of collections. As an example, here is the entry for the outfit collection for PlanetSide 2:

{
  "name": "outfit",
  "hidden": false,
  "count": "?",
  "resolve_list": ["leader", "leader_name", "leaders_stat_history", "member", "rank", "member_character",
                   "member_character_name", "member_characters_stat_history", "member_online_status"]
}

The name field contains the name of the collection. The hidden flag appears to control whether the collection is accessible to the API. If specified, count shows the size of the collection, i.e. the number of entries for that table.

Some collections also have a non-empty resolve_list, whose purpose will be covered in the Query Commands section below.

Accessing Collections

When accessing a collection without any arguments, the server will return the first entry in a collection. To see other data, you could either just return more results using the c:limit query command, but for collections with thousands of entries, this becomes infeasible rather quickly, and also puts a lot of unnecessary load on the API.

Instead, you should filter the collection for the entries we are interested in by passing any number of field/value pairs to the collection. This is done via the URL query string, which is effectively everything following the ? character in your URL.

Unique IDs

Most collections have a special field that is unique across all entries. Generally this field is named <collection>_id (i.e. character_id for the character collection, etc.), though there are some exceptions, notably fire_mode_2, which also uses fire_mode_id, and profile_2, which uses profile_id.

Additionally, the experience_rank collection has its IDs doubled-up, once for ASP and once for the regular, non-ASP ranks.

Example Queries

Here is a basic query, which returns the faction whose faction_id field has a value of 4 (i.e. NS Operatives):
https://census.daybreakgames.com/get/ps2/faction?faction_id=4

You can chain any number of field/value pairs together using the & operator. The following query only looks for items that are both knives (item_category_id=2) and specific to the TR (faction_id=3):
https://census.daybreakgames.com/get/ps2/item?item_category_id=2&faction_id=3

You may also specify multiple values for a given field by comma-separating the available options. These are then combined with a Boolean OR operator. Note that when using this syntax, the server will return multiple values without requiring the use of c:limit:
https://census.daybreakgames.com/get/ps2/world?world_id=10,13

Some collections have nested keys, like the item collection's name field, which is localized for different languages. These inner keys can be accessed using the dot (.) operator. Here is an example for querying an item by its English locale name:
https://census.daybreakgames.com/get/ps2/item?name.en=Force-Blade

Search Modifiers

Sometimes you want to match by something other than an exact match. This can be done by prefixing your value with a search modifier. Here is a list of all available search modifiers:

Character Search Modifier
< Less than
[ Less than or equal
> Greater than
] Greater than or equal
^ Starts with (RegEx)
* Contains (RegEx)
! Not equal to

You may only use one search modifier per field/value pair, but you can re-use the same field to apply multiple requirements, which are then combined with a Boolean AND operator (i.e. all filter terms need to apply before an entry is matched).

For example, this query will filter for characters that are between battle rank 15 and 30:
http://census.daybreakgames.com/get/ps2/character?battle_rank.value=]15&battle_rank.value=[30

Listing Available Fields

To be able to filter a collection by its fields necessitates knowing the names of the fields available.

Most collections will return a single entry if accessed without a query string, allowing you to view the fields returned.

Note that not all fields are populated for every collection; these optional fields are excluded from the response by default. You can append your query with ?c:includeNull=1, which will cause unpopulated fields to be set to be included, with their value set to the string "NULL".

The odd-looking c:includeNull is a query command, which will be covered in the next section.

This example also shows the fields used by the heat mechanic despite the weapon not using a heat mechanic, hence the value "NULL" in the corresponding fields: https://census.daybreakgames.com/get/ps2/weapon?c:includeNull=1

Identifying Optional Fields

There is no flag or other easy way to find out whether a given field is optional (i.e. may be "NULL" for some entries) or not.

One handy workaround is to compare the number of entries with a non-"NULL" value for a given field against the total number of entries in the collection; but it only works for collections supporting the c:has query command, which filters entries by whether the have a non-"NULL" value for a given field, and the count query verb.

Total number of weapons: http://census.daybreakgames.com/count/ps2/weapon
Weapons with a non-NULL heat_capacity value: http://census.daybreakgames.com/count/ps2/weapon?c:has=heat_capacity

If the result of these two queries is the same, the field is likely required by all entries. If it differs, it is optional.

Query commands

Query commands are special instructions that are passed on to the server as part of the URL query string. They are prefixed with c: to distinguish them from collection fields.

List of Query Commands

Query command Example Description
case c:case=0 By default, all text searches are case-sensitive. Setting this flag to False will enable a case-insensitive search. Note that using the lower-case versions of fields (e.g. character.name.first_lower) is faster than using this query command.
distinct c:distinct=field Enumerate all unique values available for the given field. Results are capped at 20'000 unique values.
exactMatchFirst c:exactMatchFirst=1 When using non-exact text searches, this will promote any exact matches to the first entry returned. This will override the sorting behaviour defined by c:sort.
has c:has=field Only include results with a non-NULL value in this field.
hide c:hide=field1,field2 Hides the fields listed from the response. show will overrule this.
includeNull c:includeNull=1 If True, NULL value fields will be included in the response. By default, these fields are excluded.
join c:join=collection Allows dynamically creating sub-queries based on the returned data. Refer to the Joined Queries section below for details.
lang c:lang=en Only return the specified locale for localised fields.
limit c:limit=10 Allows returning more than one result.
limitPerDB c:limitPerDB=10 Like c:limit but returns results on the per-database basis. This only affects the character collection, which is spread out randomly across 20 databases.
resolve c:resolve=<name> A simpler version of c:join, which allows including related data for a given collection. The list of resolvable names for a given collection can be found in the list of collections. There is no performance benefit over c:join, which is why this primer will not cover it in detail.
retry c:retry=0 By default, any failed queries will be retried once before failing server-side. Set this flag false to prevent this behaviour and fail early.
show c:show=field1,field2 Only include the fields listed in the response. This will overrule hide.
sort c:sort=field,field:-1 Sorts the response entries by the fields given. Appending a field name with :-1 will sort in descending order.
start c:start=20 Allows skipping the first X results. Useful for paginated views onto static data, be wary of dynamic tables inserting new data while you are polling using this command.
start c:start=10 Skips the given number of results. When combined with limit, this can be useful for creating page views of large datasets.
timing c:timing=1 Include query timing information in the query response. Only allowed for quests using the get query verb.
tree c:tree=field Restructures the returned data as a tree view. See below for details.

Using Tree Views

A tree view allows reformatting the returned data into a tree view by grouping the dataset by a given field's value. They are particularly helpful when working with large datasets or low-performance clients that would struggle to process the data locally.

Much like joins, covered in the next section, the tree-view command does not provide any information on error; it is simply ignored.

Keys:

Key Description
field The field to use for the tree view. The field: prefix may be omitted; the first value passed to c:tree will be interpreted as the field by default.
list Whether the given field is a list (1) or not (0).
prefix A prefix to add to the field value used in the key.
start Allows moving the tree to an inner item (see examples below).

Examples:

Joined Queries

Joined queries (aka. Joins) are used to create inner, joined queries on top of the main API query. This allows returning a large amount of related data in one batch, rather than having to send a lot of individual queries.

Vocabulary:

This section uses the following terms when talking about joins:

  • The initial non-join query sent to the API is referred to as the "top-level query".
  • The query a join is created for is its "parent". The parent may either be a top-level query or another join (see below for details).
  • The current join is called the "child".
  • It is possible to have multiple joins for the same parent. These are referred to as "siblings".
  • A join whose parent is also a join is called a "nested join".
  • An "inner" join vs. an "outer" join controls which results are included in the join. Not to be confused with the top-level query or nested joins.

Keys:

Key Description
type The collection the joined query will access. The type: prefix may be omitted; the first value passed to c:join will be interpreted as the collection name by default.
on The field in the parent query that the join is attached onto. Refer to the fields section for details.
to The field in the child collection that the join is matched to. Refer to the fields section for details.
list Whether the join is a list.
show Like c:show but uses the top-comma/single quote character for field separation (e.g. ^show:field1'field2).
hide Like c:hide but uses the top-comma/single quote character for field separation (e.g. ^hide:field1'field2).
inject_at Specifies a custom field name to insert the return data at. This will overwrite existing field names. If multiple joins use the same field name for their response, any duplicates will be suffixed with _merged.
terms Terms are used to provide a query string to the joined query. They may use non-RegEx search modifiers, i.e. all except ^ ("starts with") and * ("contains").
outer By default, all queries are treated as an "outer" join. As in SQL, this means that results will be included for joins that do not match the criteria defined via terms. You can set ^outer:0 to user "inner" join behaviour instead, which allows filtering of a parent join via a term defined for its child if both are using "inner" join behaviour. You can find examples below.

Nested and Sibling Joins

Note: When this section uses placeholders like join_1 or join_2a, these represent an entire, fully functional join as per the syntax covered above.

These placeholders are only used to keep the examples easy to follow and clean, refer to the examples section below for more working examples.

Joins can be attached to a top-level query using the c:join query command, but also to other joins. To create a join for another join, simply enclose the nested join in parenthesis and append it to the parent join:

.../parent?c:join=join_1(join_2)

Similarly, you can create multiple joins for the same parent, referred to as sibling joins here, by comma-separating the blocks:

.../parent?c:join=join_a,join_b

Note: You may also use the inject_at key to insert the data of two sibling joins into the same key. If field names collide, one of them will be appended with _merged, e.g. character_id_merged.

These two can also be nested and combined at will to create whatever structure you require:

.../parent?c:join=join_1(join_1a,join_1b),join_2(join_2a(join_2b))

Parent and Child Field Names

Joins are created by matching a field's value across two collections. By default, this matching is performed using a <collection>_id field, if present.

This means that joining characters_online_status to character works (since the parent collection's ID field, character_id, exists for both collections), while joining character to outfit_member fails because outfit_member does not have a field labelled outfit_member_id.

For mismatches like this, the field names to use may be specified using the ^on:<field> and ^to:<field> keys; "on" being the parent collection's field name and "to" being the field in the child collection.

Join Limitations

Joins generally behave much like regular queries, but there are some differences worth keeping in mind as you develop and troubleshoot your queries:

  • Joined queries do not report any errors. They are simply ignored by the parser, with the rest of the query being handled as if the join was never there.

    This also includes other joins; only the faulty join and its children will be compromised by the error, previously processed joins will still take full effect.

  • Joins may not use the RegEx search modifiers ^ ("starts with") or * ("contains") as part of their terms. Doing so will invalidate the entire join.

  • The parent collection's field must be included in the response for a join to function. This is due to the join being processed after the parent query has executed.

    You can hide the child's "to" field in the join, though, since the join has already been processed by the time its response is generated.

  • Update: As of May 2023, the parsing logic has been updated to no longer have this limitation.

    It is not possible to create new nested sibling joins after you added a grandchild join.

    In the example below, the presence of Delta breaks Charlie and all of its children or other siblings.

             Top-level Query
               |        
         Alpha (Level 1)
          |           \
 Bravo (Level 2)   Charlie (Level 2, IGNORED) 
          |            |
 Delta (Level 3)   Epsilon (Level 3, IGNORED)

    Note that this issue only affects nested joins. A sibling of Alpha (i.e. another Level 1 join) would still work correctly as it is joined directly to a top-level query.

    Possible workarounds:

    • Removal of Delta would fix Charlie.

    • The last sibling (Charlie in this case) could still have grandchildren as there are no more siblings after it.

    • Eliminating the Alpha join by querying the table Alpha joined to also resolves this issue as Bravo and Charlie are no longer nested joins.

    • It is possible to join the exact same table more than once. In this case, one could join Alpha once with its Bravo sub-tree, and once with its Charlie sub-tree. Inserting the two under the same key will merge the result payloads, including any joins.

      If you pursue this workaround, be sure to hide all but the essential fields as any duplicates will be inserted as separate fields, named <fieldname>_merged, which could double your payload size.

Example Joins

This Wiki only contains general Census API topics, development notes and other version-independent documentation.
Visit Read the Docs for usage examples and the full API reference.

Documentation

Developer Reference

Clone this wiki locally