Skip to content

Importer API

flori edited this page Nov 15, 2026 · 1 revision

How to write an import module: the piece of code that lets FAuction import the data of another auction house plugin. Administrators looking for how to run an import should read Importing Data.

A module only reads its source and turns each row into a simple record. Everything delicate stays in FAuction and is written once for all modules: validation, writes to the database in transactional batches, protection against duplicates across runs, threads, cancellation, report, command.

The package fr.florianpal.fauction.api.importer (with its event and testing sub-packages) is the only part of FAuction whose compatibility is guaranteed. Everything else is internal.

Setup

Same Maven repository as the API:

<dependency>
    <groupId>fr.florianpal</groupId>
    <artifactId>fauction</artifactId>
    <version>2.4.0</version>
    <scope>provided</scope>
</dependency>

plugin.yml of your plugin:

depend: [FAuction]

Register the module when your plugin enables. The registry is published in the Bukkit ServicesManager (or FAuction.getApi().getImporterRegistry()):

public static void onEnable(Plugin plugin) {
    ImporterRegistry registry = Bukkit.getServicesManager().load(ImporterRegistry.class);
    if (registry == null) {
        plugin.getLogger().warning("FAuction is not enabled, the import module is not registered");
        return;
    }
    registry.register(plugin, new MyYamlImporter());
    // Nothing to do in onDisable : FAuction drops the modules of a plugin when it is disabled.
}

register throws an IllegalArgumentException when the id is invalid (1 to 32 characters among [a-z0-9_-]), already taken, or when the module needs a newer API than this FAuction provides.

The contract

DataImporter — what you implement

Method
String id() Unique, used on the command line: /ah admin import run <id>.
String displayName() Shown by /ah admin import list; put the tested version of the source in it.
Set<ImportDataType> supportedTypes() Among AUCTION, EXPIRED, HISTORIC, PENDING_CURRENCY.
ImportAvailability checkAvailability(ImportContext) Whether the source can be read now. Refuse while the source plugin is enabled.
void read(ImportContext, ImportSink) Reads the source and pushes every row into the sink. Runs off the main thread.
int requiredApiVersion() Override it with return ImporterApi.API_VERSION;, see Versioning.

The records

Record Lands in Mandatory Optional (null)
ImportedAuction the market sourceId, sellerUuid, item, price (> 0), listedAt sellerName, currency
ImportedExpired the owner's /ah expire sourceId, ownerUuid, item, price (≥ 0, display only), listedAt ownerName
ImportedHistoric the seller's history sourceId, sellerUuid, buyerUuid, item, price (> 0) sellerName, buyerName, listedAt, soldAt
ImportedPendingCurrency paid when the player is online sourceId, playerUuid, sourceCurrency, amount (> 0) reason

Each record has a builder (ImportedAuction.builder()...build()) that throws a NullPointerException when a mandatory field is missing.

  • sourceId: a stable identifier of the row in the source (SQL id, YAML key, UUID...), at most 191 characters, unique per type. Two reads of the same source must give the same ids: that is what lets an administrator run the import again without duplicating anything. If the source renumbers its ids, use a hash of the content (see the NexusAuctionHouse module).
  • Names: optional everywhere. When absent (or made only of colour codes), FAuction resolves them from the UUID. Colour codes are removed and the name is cut to 36 characters: pass the raw value.
  • Dates: a date in the future is brought back to now. A missing soldAt falls back on listedAt, then on the date of the import.
  • Currencies: give the name of the currency on the source side ("Vault", "CoinsEngine-coins"...). FAuction converts it with import.currency-map of its config.yml; never convert it yourself. ImportedAuction.currency is null for the main currency of the server.

ImportedItem — the item, in the form the source stores it

Factory For
ImportedItem.of(ItemStack) an item you already built
ofBukkitBase64(String) / ofBukkitBytes(byte[]) BukkitObjectOutputStream (most Spigot plugins). The base64 is read with the MIME decoder: line breaks of older encoders are fine.
ofPaperBase64(String) / ofPaperBytes(byte[]) ItemStack.serializeAsBytes() of Paper
ofVanillaNbtGzip(byte[]) vanilla NBT compressed with GZIP, with its DataVersion (AxAPI)

The factories check the shape of the data right away (empty, invalid base64, no GZIP header) and throw an IllegalArgumentException. The decoding itself is done by FAuction: an item that cannot be decoded refuses its row only.

ImportSink — where you push

accept(...) for each record type, skip(sourceId, reason) for a row you cannot or will not convert, and isCancelled() to test in your loops. Rows of a type the administrator did not ask for (--types) are ignored by the sink; context.requestedTypes() lets you skip reading them.

ImportContext and ImportHelpers

ImportContext
logger() prefixed [Import:<id>]
options() the --opt key=value of the command, keys in lower case
dryRun(), requestedTypes(), convertBids() the options of the run
clock() read "now" from it, so your tests can fix the date
pluginsFolder(), sourcePlugin(name) the plugins/ folder, the source plugin if loaded
helpers() see below
ImportHelpers
openJdbc(url, user, password) a read-only connection to the source database (a SQLite file of the source works)
sourceDataFolder(pluginName) plugins/<pluginName>/, loaded or not
loadYaml(file) absent file = empty configuration
loadJson(file) / loadJson(file, type) with the Gson of the server; absent or empty file = JsonNull / null; invalid JSON = IOException
resolvePlayerName(uuid) the name the server knows, or the UUID
lastModified(file) when the source last wrote that file: evaluate the expirations at that instant, not "now" (see Rules)
decodeItem(importedItem) the ItemStack, for a module that has to look into an item (split a partially sold stack...)
fromEpochMillis, fromEpochSeconds Instant conversions
callSync(callable) runs on the main thread and waits, for the Bukkit API calls that need it

Rules

  • Never write to the source. Read only.
  • Refuse to run while the source plugin is enabled (checkAvailability).
  • One stable sourceId per row and per type.
  • Test isCancelled() in your loops.
  • skip() a bad row, don't throw: an exception out of read() stops the whole import (the rows pushed before are still imported).
  • Evaluate time-dependent states at a fixed instant: whether a sale is still on the market or expired, whether an auction is running or ended, must be decided at helpers().lastModified(sourceFile) (when the source stopped), never at clock().instant(). Otherwise a second run a few days later sees the same sale as expired and imports its item a second time. (FAuction also treats AUCTION and EXPIRED rows as sharing their sourceIds, as a safety net.)
  • Never lose an item nor money: when a state of the source has no equivalent in FAuction, give the item back to its owner (ImportedExpired) and the money to whom it belongs (ImportedPendingCurrency), or skip() with a reason so the administrator can act.
  • read() runs off the main thread: use helpers().callSync(...) for the Bukkit API calls that require it.

Complete example

A module for an imaginary plugin "MyAH" that keeps its sales in plugins/MyAH/listings.yml. AbstractYamlImporter does the loop: every child of a root section is a row.

public class MyYamlImporter extends AbstractYamlImporter {

    @Override
    public String id() {
        return "myah";
    }

    @Override
    public String displayName() {
        return "MyAH 1.0";
    }

    @Override
    public Set<ImportDataType> supportedTypes() {
        return EnumSet.of(ImportDataType.AUCTION, ImportDataType.EXPIRED);
    }

    @Override
    public int requiredApiVersion() {
        return ImporterApi.API_VERSION;
    }

    @Override
    public ImportAvailability checkAvailability(ImportContext context) {
        // The source must be stopped : it would keep selling the items being imported.
        if (context.sourcePlugin("MyAH").map(plugin -> plugin.isEnabled()).orElse(false)) {
            return ImportAvailability.unavailable("MyAH is still enabled, remove it first");
        }
        File file = file(context);
        return file.isFile()
                ? ImportAvailability.available("Found " + file.getPath())
                : ImportAvailability.unavailable("No " + file.getPath());
    }

    @Override
    protected File file(ImportContext context) {
        return new File(context.helpers().sourceDataFolder("MyAH"), "listings.yml");
    }

    @Override
    protected List<YamlSource<?>> sources(ImportContext context) {
        return List.of(
                new YamlSource<>(ImportDataType.AUCTION, "listings", (key, section, ctx) -> {
                    String seller = section.getString("seller");
                    if (seller == null) {
                        // Kept in the report with its reason, the import goes on.
                        throw new ImportSkipException(key, "No seller");
                    }
                    return new ImportedAuction(
                            key,                                              // stable sourceId
                            UUID.fromString(seller),
                            null,                                             // FAuction resolves the name
                            ImportedItem.ofBukkitBase64(section.getString("item")),
                            section.getDouble("price"),
                            Instant.ofEpochMilli(section.getLong("created")));
                }),
                new YamlSource<>(ImportDataType.EXPIRED, "expired", (key, section, ctx) ->
                        new ImportedExpired(key, UUID.fromString(section.getString("owner")), null,
                                ImportedItem.ofBukkitBase64(section.getString("item")), 0, ctx.clock().instant())));
    }
}

AbstractSqlImporter is the same for a database: give the connection, one query per type and a RowMapper:

@Override
protected Connection openConnection(ImportContext context) throws Exception {
    return context.helpers().openJdbc(context.options().get("url"),
            context.options().getOrDefault("user", ""), context.options().getOrDefault("password", ""));
}

@Override
protected List<SqlSource<?>> sources(ImportContext context) {
    return List.of(
            new SqlSource<>(ImportDataType.HISTORIC,
                    "SELECT id, seller, buyer, item, price, sold_at FROM sales ORDER BY id",
                    (row, ctx) -> new ImportedHistoric(
                            "sale-" + row.getLong("id"),
                            UUID.fromString(row.getString("seller")), null,
                            UUID.fromString(row.getString("buyer")), null,
                            ImportedItem.ofBukkitBase64(row.getString("item")),
                            row.getDouble("price"),
                            null,                                         // listing date not kept
                            Instant.ofEpochSecond(row.getLong("sold_at")))),
            new SqlSource<>(ImportDataType.PENDING_CURRENCY,
                    "SELECT player, amount FROM unpaid",
                    (row, ctx) -> new ImportedPendingCurrency(
                            "unpaid-" + row.getString("player"),
                            UUID.fromString(row.getString("player")),
                            "Vault",                                      // mapped by import.currency-map
                            row.getDouble("amount"),
                            "Sale not collected")));
}

In both base classes, a mapper that throws ImportSkipException skips its row with that reason; any other exception skips the row with the message of the exception, except an SQLException, which stops the import (the connection or the query broke, every next row would fail the same way). A mapper producing a row of another type than its source declares is skipped too. supportedTypes() is declared by the module.

For a JSON source, implement DataImporter directly: the Auction-House module is the reference (a single notes.json, one row per note, UUID identifiers):

JsonElement root = context.helpers().loadJson(notesFile);
JsonArray notes = root.getAsJsonArray();
for (int index = 0; index < notes.size(); index++) {
    if (sink.isCancelled()) {
        return;
    }
    JsonObject note = notes.get(index).getAsJsonObject();
    String sourceId = note.get("noteID").getAsString();
    try {
        // ... sink.accept(new ImportedAuction(sourceId, ...)) following the state of the note
    } catch (RuntimeException e) {
        sink.skip(sourceId, "Unreadable note : " + e.getMessage());
    }
}

Testing your module

fr.florianpal.fauction.api.importer.testing.RecordingImportSink records what your module pushes, without FAuction nor a database:

RecordingImportSink sink = new RecordingImportSink();
new MyYamlImporter().read(context, sink);

assertEquals(1, sink.auctions().size());
assertEquals("17", sink.auctions().get(0).sourceId());
assertEquals("No seller", sink.skips().get(0).reason());

cancelAfter(n) cancels by itself after n rows, to test your cancellation. For the context, use fr.florianpal.fauction.api.importer.testing.TestImportContext rather than implementing ImportContext yourself (methods may be added to it):

TestImportContext context = new TestImportContext(tempPluginsFolder)   // clock fixed at TestImportContext.NOW
        .option("url", "jdbc:...")                                       // --opt url=...
        .types(EnumSet.of(ImportDataType.AUCTION))                       // --types auction
        .plugin("MyAH", mockedSourcePlugin);                             // to test the refusal while enabled

Events

In fr.florianpal.fauction.api.importer.event:

  • ImportStartEvent (cancellable): an import is about to start. getImporterId(), getTypes(), isDryRun(). Cancelling it prevents anything from being read.
  • ImportFinishEvent: an import finished, whatever its outcome. getReport() gives the ImportReport (status, counters per type, refused rows with their reason, path of the detailed report). Fired from the import thread, so usually asynchronously.

No AuctionAddEvent nor ExpireAddEvent is fired for the imported rows.

Versioning

ImporterApi.API_VERSION is the version of the API this FAuction provides (currently 1). A module whose requiredApiVersion() is higher is refused at registration with a message asking to update FAuction. Implement it as return ImporterApi.API_VERSION;: the constant is copied into your module when it is compiled, so it always states the version you built against.

Compatibility policy: a new data type (bids, once FAuction has them) is added as a new ImportDataType constant, a new record and a new default method of ImportSink, so existing modules keep compiling and running; API_VERSION is then increased. The signature of an existing record never changes.

Clone this wiki locally