Repository navigation
Importer API
How to write an import module: the piece of code that lets FAuction import the data of another auction house plugin. Administrators looking for how to run an import should read Importing Data.
A module only reads its source and turns each row into a simple record. Everything delicate stays in FAuction and is written once for all modules: validation, writes to the database in transactional batches, protection against duplicates across runs, threads, cancellation, report, command.
The package fr.florianpal.fauction.api.importer (with its event and testing sub-packages) is the only part of FAuction whose compatibility is guaranteed. Everything else is internal.
Same Maven repository as the API:
<dependency>
<groupId>fr.florianpal</groupId>
<artifactId>fauction</artifactId>
<version>2.4.0</version>
<scope>provided</scope>
</dependency>plugin.yml of your plugin:
depend: [FAuction]Register the module when your plugin enables. The registry is published in the Bukkit ServicesManager (or FAuction.getApi().getImporterRegistry()):
public static void onEnable(Plugin plugin) {
ImporterRegistry registry = Bukkit.getServicesManager().load(ImporterRegistry.class);
if (registry == null) {
plugin.getLogger().warning("FAuction is not enabled, the import module is not registered");
return;
}
registry.register(plugin, new MyYamlImporter());
// Nothing to do in onDisable : FAuction drops the modules of a plugin when it is disabled.
}register throws an IllegalArgumentException when the id is invalid (1 to 32 characters among [a-z0-9_-]), already taken, or when the module needs a newer API than this FAuction provides.
| Method | |
|---|---|
String id() |
Unique, used on the command line: /ah admin import run <id>. |
String displayName() |
Shown by /ah admin import list; put the tested version of the source in it. |
Set<ImportDataType> supportedTypes() |
Among AUCTION, EXPIRED, HISTORIC, PENDING_CURRENCY. |
ImportAvailability checkAvailability(ImportContext) |
Whether the source can be read now. Refuse while the source plugin is enabled. |
void read(ImportContext, ImportSink) |
Reads the source and pushes every row into the sink. Runs off the main thread. |
int requiredApiVersion() |
Override it with return ImporterApi.API_VERSION;, see Versioning. |
| Record | Lands in | Mandatory | Optional (null) |
|---|---|---|---|
ImportedAuction |
the market |
sourceId, sellerUuid, item, price (> 0), listedAt
|
sellerName, currency
|
ImportedExpired |
the owner's /ah expire
|
sourceId, ownerUuid, item, price (≥ 0, display only), listedAt
|
ownerName |
ImportedHistoric |
the seller's history |
sourceId, sellerUuid, buyerUuid, item, price (> 0) |
sellerName, buyerName, listedAt, soldAt
|
ImportedPendingCurrency |
paid when the player is online |
sourceId, playerUuid, sourceCurrency, amount (> 0) |
reason |
Each record has a builder (ImportedAuction.builder()...build()) that throws a NullPointerException when a mandatory field is missing.
-
sourceId: a stable identifier of the row in the source (SQL id, YAML key, UUID...), at most 191 characters, unique per type. Two reads of the same source must give the same ids: that is what lets an administrator run the import again without duplicating anything. If the source renumbers its ids, use a hash of the content (see the NexusAuctionHouse module). - Names: optional everywhere. When absent (or made only of colour codes), FAuction resolves them from the UUID. Colour codes are removed and the name is cut to 36 characters: pass the raw value.
-
Dates: a date in the future is brought back to now. A missing
soldAtfalls back onlistedAt, then on the date of the import. -
Currencies: give the name of the currency on the source side (
"Vault","CoinsEngine-coins"...). FAuction converts it withimport.currency-mapof itsconfig.yml; never convert it yourself.ImportedAuction.currencyisnullfor the main currency of the server.
| Factory | For |
|---|---|
ImportedItem.of(ItemStack) |
an item you already built |
ofBukkitBase64(String) / ofBukkitBytes(byte[])
|
BukkitObjectOutputStream (most Spigot plugins). The base64 is read with the MIME decoder: line breaks of older encoders are fine. |
ofPaperBase64(String) / ofPaperBytes(byte[])
|
ItemStack.serializeAsBytes() of Paper |
ofVanillaNbtGzip(byte[]) |
vanilla NBT compressed with GZIP, with its DataVersion (AxAPI) |
The factories check the shape of the data right away (empty, invalid base64, no GZIP header) and throw an IllegalArgumentException. The decoding itself is done by FAuction: an item that cannot be decoded refuses its row only.
accept(...) for each record type, skip(sourceId, reason) for a row you cannot or will not convert, and isCancelled() to test in your loops. Rows of a type the administrator did not ask for (--types) are ignored by the sink; context.requestedTypes() lets you skip reading them.
ImportContext |
|
|---|---|
logger() |
prefixed [Import:<id>]
|
options() |
the --opt key=value of the command, keys in lower case |
dryRun(), requestedTypes(), convertBids()
|
the options of the run |
clock() |
read "now" from it, so your tests can fix the date |
pluginsFolder(), sourcePlugin(name)
|
the plugins/ folder, the source plugin if loaded |
helpers() |
see below |
ImportHelpers |
|
|---|---|
openJdbc(url, user, password) |
a read-only connection to the source database (a SQLite file of the source works) |
sourceDataFolder(pluginName) |
plugins/<pluginName>/, loaded or not |
loadYaml(file) |
absent file = empty configuration |
loadJson(file) / loadJson(file, type)
|
with the Gson of the server; absent or empty file = JsonNull / null; invalid JSON = IOException
|
resolvePlayerName(uuid) |
the name the server knows, or the UUID |
lastModified(file) |
when the source last wrote that file: evaluate the expirations at that instant, not "now" (see Rules) |
decodeItem(importedItem) |
the ItemStack, for a module that has to look into an item (split a partially sold stack...) |
fromEpochMillis, fromEpochSeconds
|
Instant conversions |
callSync(callable) |
runs on the main thread and waits, for the Bukkit API calls that need it |
- Never write to the source. Read only.
-
Refuse to run while the source plugin is enabled (
checkAvailability). -
One stable
sourceIdper row and per type. -
Test
isCancelled()in your loops. -
skip()a bad row, don't throw: an exception out ofread()stops the whole import (the rows pushed before are still imported). -
Evaluate time-dependent states at a fixed instant: whether a sale is still on the market or expired, whether an auction is running or ended, must be decided at
helpers().lastModified(sourceFile)(when the source stopped), never atclock().instant(). Otherwise a second run a few days later sees the same sale as expired and imports its item a second time. (FAuction also treatsAUCTIONandEXPIREDrows as sharing theirsourceIds, as a safety net.) -
Never lose an item nor money: when a state of the source has no equivalent in FAuction, give the item back to its owner (
ImportedExpired) and the money to whom it belongs (ImportedPendingCurrency), orskip()with a reason so the administrator can act. -
read()runs off the main thread: usehelpers().callSync(...)for the Bukkit API calls that require it.
A module for an imaginary plugin "MyAH" that keeps its sales in plugins/MyAH/listings.yml. AbstractYamlImporter does the loop: every child of a root section is a row.
public class MyYamlImporter extends AbstractYamlImporter {
@Override
public String id() {
return "myah";
}
@Override
public String displayName() {
return "MyAH 1.0";
}
@Override
public Set<ImportDataType> supportedTypes() {
return EnumSet.of(ImportDataType.AUCTION, ImportDataType.EXPIRED);
}
@Override
public int requiredApiVersion() {
return ImporterApi.API_VERSION;
}
@Override
public ImportAvailability checkAvailability(ImportContext context) {
// The source must be stopped : it would keep selling the items being imported.
if (context.sourcePlugin("MyAH").map(plugin -> plugin.isEnabled()).orElse(false)) {
return ImportAvailability.unavailable("MyAH is still enabled, remove it first");
}
File file = file(context);
return file.isFile()
? ImportAvailability.available("Found " + file.getPath())
: ImportAvailability.unavailable("No " + file.getPath());
}
@Override
protected File file(ImportContext context) {
return new File(context.helpers().sourceDataFolder("MyAH"), "listings.yml");
}
@Override
protected List<YamlSource<?>> sources(ImportContext context) {
return List.of(
new YamlSource<>(ImportDataType.AUCTION, "listings", (key, section, ctx) -> {
String seller = section.getString("seller");
if (seller == null) {
// Kept in the report with its reason, the import goes on.
throw new ImportSkipException(key, "No seller");
}
return new ImportedAuction(
key, // stable sourceId
UUID.fromString(seller),
null, // FAuction resolves the name
ImportedItem.ofBukkitBase64(section.getString("item")),
section.getDouble("price"),
Instant.ofEpochMilli(section.getLong("created")));
}),
new YamlSource<>(ImportDataType.EXPIRED, "expired", (key, section, ctx) ->
new ImportedExpired(key, UUID.fromString(section.getString("owner")), null,
ImportedItem.ofBukkitBase64(section.getString("item")), 0, ctx.clock().instant())));
}
}AbstractSqlImporter is the same for a database: give the connection, one query per type and a RowMapper:
@Override
protected Connection openConnection(ImportContext context) throws Exception {
return context.helpers().openJdbc(context.options().get("url"),
context.options().getOrDefault("user", ""), context.options().getOrDefault("password", ""));
}
@Override
protected List<SqlSource<?>> sources(ImportContext context) {
return List.of(
new SqlSource<>(ImportDataType.HISTORIC,
"SELECT id, seller, buyer, item, price, sold_at FROM sales ORDER BY id",
(row, ctx) -> new ImportedHistoric(
"sale-" + row.getLong("id"),
UUID.fromString(row.getString("seller")), null,
UUID.fromString(row.getString("buyer")), null,
ImportedItem.ofBukkitBase64(row.getString("item")),
row.getDouble("price"),
null, // listing date not kept
Instant.ofEpochSecond(row.getLong("sold_at")))),
new SqlSource<>(ImportDataType.PENDING_CURRENCY,
"SELECT player, amount FROM unpaid",
(row, ctx) -> new ImportedPendingCurrency(
"unpaid-" + row.getString("player"),
UUID.fromString(row.getString("player")),
"Vault", // mapped by import.currency-map
row.getDouble("amount"),
"Sale not collected")));
}In both base classes, a mapper that throws ImportSkipException skips its row with that reason; any other exception skips the row with the message of the exception, except an SQLException, which stops the import (the connection or the query broke, every next row would fail the same way). A mapper producing a row of another type than its source declares is skipped too. supportedTypes() is declared by the module.
For a JSON source, implement DataImporter directly: the Auction-House module is the reference (a single notes.json, one row per note, UUID identifiers):
JsonElement root = context.helpers().loadJson(notesFile);
JsonArray notes = root.getAsJsonArray();
for (int index = 0; index < notes.size(); index++) {
if (sink.isCancelled()) {
return;
}
JsonObject note = notes.get(index).getAsJsonObject();
String sourceId = note.get("noteID").getAsString();
try {
// ... sink.accept(new ImportedAuction(sourceId, ...)) following the state of the note
} catch (RuntimeException e) {
sink.skip(sourceId, "Unreadable note : " + e.getMessage());
}
}fr.florianpal.fauction.api.importer.testing.RecordingImportSink records what your module pushes, without FAuction nor a database:
RecordingImportSink sink = new RecordingImportSink();
new MyYamlImporter().read(context, sink);
assertEquals(1, sink.auctions().size());
assertEquals("17", sink.auctions().get(0).sourceId());
assertEquals("No seller", sink.skips().get(0).reason());cancelAfter(n) cancels by itself after n rows, to test your cancellation. For the context, use fr.florianpal.fauction.api.importer.testing.TestImportContext rather than implementing ImportContext yourself (methods may be added to it):
TestImportContext context = new TestImportContext(tempPluginsFolder) // clock fixed at TestImportContext.NOW
.option("url", "jdbc:...") // --opt url=...
.types(EnumSet.of(ImportDataType.AUCTION)) // --types auction
.plugin("MyAH", mockedSourcePlugin); // to test the refusal while enabledIn fr.florianpal.fauction.api.importer.event:
-
ImportStartEvent(cancellable): an import is about to start.getImporterId(),getTypes(),isDryRun(). Cancelling it prevents anything from being read. -
ImportFinishEvent: an import finished, whatever its outcome.getReport()gives theImportReport(status, counters per type, refused rows with their reason, path of the detailed report). Fired from the import thread, so usually asynchronously.
No AuctionAddEvent nor ExpireAddEvent is fired for the imported rows.
ImporterApi.API_VERSION is the version of the API this FAuction provides (currently 1). A module whose requiredApiVersion() is higher is refused at registration with a message asking to update FAuction. Implement it as return ImporterApi.API_VERSION;: the constant is copied into your module when it is compiled, so it always states the version you built against.
Compatibility policy: a new data type (bids, once FAuction has them) is added as a new ImportDataType constant, a new record and a new default method of ImportSink, so existing modules keep compiling and running; API_VERSION is then increased. The signature of an existing record never changes.