Skip to content

Implementing multiple destinations for mcopy donwloads #641

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Sep 26, 2025
Merged

Implementing multiple destinations for mcopy donwloads #641

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
a01a24b
Bump devcontainer to Java 21
Crafter-Y Sep 23, 2025
a00c425
itzg/bungeecord moved to itzg/docker-mc-proxy
Crafter-Y Sep 23, 2025
7d01a80
implementing @ delimiter and adding unit tests
Crafter-Y Sep 23, 2025
03ac397
update documentation
Crafter-Y Sep 23, 2025
c317f64
making --to parameter required again and used as base directory for r…
Crafter-Y Sep 23, 2025
90ee1cb
making manifests work for different destinations
Crafter-Y Sep 23, 2025
9701aad
updated docs
Crafter-Y Sep 23, 2025
25dc707
most of the requested changes
Crafter-Y Sep 25, 2025
38a1b93
implemented < as delimiter
Crafter-Y Sep 25, 2025
8609830
Create destination directory reactively
itzg Sep 26, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .devcontainer/devcontainer.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
{
"name": "Java",
// Or use a Dockerfile or Docker Compose file. More info: https://containers.dev/guide/dockerfile
"image": "mcr.microsoft.com/devcontainers/java:17",
"image": "mcr.microsoft.com/devcontainers/java:21",

"features": {
"ghcr.io/devcontainers/features/java:1": {
Expand Down
143 changes: 80 additions & 63 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,78 +2,92 @@
[![test](https://github.com/itzg/mc-image-helper/actions/workflows/test.yml/badge.svg)](https://github.com/itzg/mc-image-helper/actions/workflows/test.yml)
![LOC](https://img.shields.io/endpoint?url=https%3A%2F%2Fshields-codetab-code-loc-bridge.vercel.app%2Fapi%2Fcodeloc%3Fgithub%3Ditzg%2Fmc-image-helper%26language%3Djava)

This tool does the complicated bits for the [itzg/minecraft-server](https://github.com/itzg/docker-minecraft-server) and [itzg/bungeecord](https://github.com/itzg/docker-bungeecord/) Docker images.
This tool does the complicated bits for the [itzg/minecraft-server](https://github.com/itzg/docker-minecraft-server) and [itzg/docker-mc-proxy](https://github.com/itzg/docker-mc-proxy) Docker images.

## Usage

> **NOTE** The following documentation may not always be up-to-date. Please be sure to use `-h` or `--help` after any subcommand to view the current usage.

```
Usage: mc-image-helper [-hs] [--debug] [COMMAND]
--debug Enable debug output. Can also set environment variable
DEBUG_HELPER
-h, --help Show this usage and exit
-s, --silent Don't output logs even if there's an error
Usage: mc-image-helper [-hsV] [--debug | --logging=<loggingLevel>] [COMMAND]
--debug Enable debug output. Can also set environment variables
DEBUG_HELPER or DEBUG
-h, --help Show this usage and exit
--logging=<loggingLevel>
Set logging to specific level.
Valid values:
-s, --silent Don't output logs even if there's an error
-V, --version
Commands:
asciify Converts UTF-8 on stdin to ASCII by escaping
Unicode characters
assert Provides assertion operators for verifying
container setup
compare-versions Used for shell scripting, exits with success(0)
when comparison is satisfied or 1 when not
curseforge-files Download and manage individual mod/plugin files
from CurseForge
find Specialized replacement for GNU's find
get Download a file
asciify Converts UTF-8 on stdin to ASCII by escaping
Unicode characters
assert Provides assertion operators for verifying
container setup
compare-versions Used for shell scripting, exits with success
(0) when comparison is satisfied or 1 when
not
curseforge-files Download and manage individual mod/plugin
files from CurseForge
find Specialized replacement for GNU's find
get Download a file
github
hash Outputs an MD5 hash of the standard input
install-curseforge Downloads, installs, and upgrades CurseForge
modpacks
install-fabric-loader Provides a few ways to obtain a Fabric loader with
simple cleanup of previous loader instances
install-forge Downloads and installs a requested version of Forge
install-modrinth-modpack Supports installation of Modrinth modpacks along
with the associated mod loader
install-neoforge Downloads and installs a requested version of
NeoForge
install-paper Installs selected PaperMC
install-purpur Downloads latest or selected version of Purpur
install-quilt Installs Quilt mod loader
interpolate Interpolates existing files in one or more
directories
java-release Outputs the Java release number, such as 8, 11, 17
hash Outputs an MD5 hash of the standard input
install-curseforge Downloads, installs, and upgrades CurseForge
modpacks
install-fabric-loader Provides a few ways to obtain a Fabric loader
with simple cleanup of previous loader
instances
install-forge Downloads and installs a requested version of
Forge
install-modrinth-modpack Supports installation of Modrinth modpacks
along with the associated mod loader
install-neoforge Downloads and installs a requested version of
NeoForge
install-paper Installs selected PaperMC
install-purpur Downloads latest or selected version of Purpur
install-quilt Installs Quilt mod loader
interpolate Interpolates existing files in one or more
directories
java-release Outputs the Java release number, such as 8,
11, 17
manage-users
maven-download Downloads a maven artifact from a Maven repository
modrinth Automates downloading of modrinth resources
mcopy Multi-source file copy operation with with managed
cleanup. Supports auto-detected sourcing from
file list, directories, and URLs
network-interfaces Provides simple operations to list network
interface names and check existence
patch Patches one or more existing files using JSON path
based operations
Supports the file formats:
- JSON
- JSON5
- Yaml
- TOML, but processed output is not pretty
resolve-minecraft-version Resolves and validate latest, snapshot, and
specific versions
set-properties Maps environment variables to a properties file
show-all-subcommand-usage Renders all of the subcommand usage as markdown
sections for README
sync Synchronizes the contents of one directory to
another.
sync-and-interpolate Synchronizes the contents of one directory to
another with conditional variable interpolation.
maven-download Downloads a maven artifact from a Maven
repository
modrinth Automates downloading of modrinth resources
mcopy Multi-source file copy operation with with
managed cleanup. Supports auto-detected
sourcing from file list, directories, and
URLs
network-interfaces Provides simple operations to list network
interface names and check existence
patch Patches one or more existing files using JSON
path based operations
Supports the file formats:
- JSON
- JSON5
- Yaml
- TOML, but processed output is not pretty
resolve-minecraft-version Resolves and validate latest, snapshot, and
specific versions
set-properties Maps environment variables to a properties
file
show-all-subcommand-usage Renders all of the subcommand usage as
markdown sections for README
sync Synchronizes the contents of one directory to
another.
sync-and-interpolate Synchronizes the contents of one directory to
another with conditional variable
interpolation.
test-logging-levels
toml-path Extracts a path from a TOML file using json-path
syntax
yaml-path Extracts a path from a YAML file using json-path
syntax
vanillatweaks Downloads Vanilla Tweaks resource packs, data
packs, or crafting tweaks given a share code or
pack file
toml-path Extracts a path from a TOML file using
json-path syntax
vanillatweaks Downloads Vanilla Tweaks resource packs, data
packs, or crafting tweaks given a share
code or pack file
version-from-modrinth-projects Finds a compatible Minecraft version across
given Modrinth projects
yaml-path Extracts a path from a YAML file using
json-path syntax
```

For [patch](#patch) command [see below](#patch-schemas) for a description of [PatchSet](#patchset) and [PatchDefinition](#patchdefinition) JSON schemas.
Expand Down Expand Up @@ -851,7 +865,7 @@ Downloads a maven artifact from a Maven repository
### mcopy

```
Usage: mc-image-helper mcopy [-hz] [--file-is-listing]
Usage: mc-image-helper mcopy [-hz] [--file-is-listing]
[--ignore-missing-sources] [--quiet-when-skipped]
[--skip-existing] [--glob=GLOB]
[--scope=<manifestId>] --to=<dest> SRC[,
Expand All @@ -860,6 +874,8 @@ Multi-source file copy operation with with managed cleanup. Supports
auto-detected sourcing from file list, directories, and URLs
SRC[,|<nl>SRC...]... Any mix of source file, directory, or URLs.
Can be optionally comma or newline separated.
Per-file destinations can be assigned by
destination<source
--file-is-listing Source files or URLs are processed as a line
delimited list of sources.
For remote listing files, the contents must all be
Expand All @@ -879,6 +895,7 @@ auto-detected sourcing from file list, directories, and URLs
--to, --output-directory=<dest>

-z, --skip-up-to-date

```

### modrinth
Expand Down
85 changes: 59 additions & 26 deletions src/main/java/me/itzg/helpers/sync/MulitCopyCommand.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
import me.itzg.helpers.errors.GenericException;
import me.itzg.helpers.errors.InvalidParameterException;
import me.itzg.helpers.files.Manifests;
import me.itzg.helpers.files.ReactiveFileUtils;
import me.itzg.helpers.http.FailedRequestException;
import me.itzg.helpers.http.Fetch;
import me.itzg.helpers.http.SharedFetch;
Expand Down Expand Up @@ -74,18 +75,18 @@ public class MulitCopyCommand implements Callable<Integer> {
paramLabel = "SRC",
description = "Any mix of source file, directory, or URLs."
+ "%nCan be optionally comma or newline separated."
+ "%nPer-file destinations can be assigned by destination<source"
)
List<String> sources;

private final static String destinationDelimiter = "<";

@Override
public Integer call() throws Exception {

Files.createDirectories(dest);

Flux.fromIterable(sources)
.map(String::trim)
.filter(s -> !s.isEmpty())
.flatMap(source -> processSource(source, fileIsListingOption))
.flatMap(source -> processSource(source, fileIsListingOption, dest))
.collectList()
.flatMap(this::cleanupAndSaveManifest)
.block();
Expand Down Expand Up @@ -113,24 +114,55 @@ private Mono<?> cleanupAndSaveManifest(List<Path> paths) {
});
}

private Publisher<Path> processSource(String source, boolean fileIsListing) {
if (Uris.isUri(source)) {
return fileIsListing ? processRemoteListingFile(source) : processRemoteSource(source);
} else {
final Path path = Paths.get(source);
if (!Files.exists(path)) {
throw new GenericException(String.format("Source file '%s' does not exist", source));
}
private Publisher<Path> processSource(String source, boolean fileIsListing, Path parentDestination) {
final Path destination;
final String resolvedSource;

if (Files.isDirectory(path)) {
return processDirectory(path);
final int delimiterPos = source.indexOf(destinationDelimiter);
if (delimiterPos > 0) {
destination = parentDestination.resolve(Paths.get(source.substring(0, delimiterPos)));
resolvedSource = source.substring(delimiterPos + 1);
}
else {
destination = parentDestination;
resolvedSource = source;
}

if (fileIsListing) {
if (Uris.isUri(resolvedSource)) {
return processRemoteListingFile(resolvedSource, destination);
} else {
return fileIsListing ? processListingFile(path) : processFile(path);
final Path path = Paths.get(resolvedSource);
if (Files.isDirectory(path)) {
throw new GenericException(String.format("Specified listing file '%s' is a directory", resolvedSource));
}
if (!Files.exists(path)) {
throw new GenericException(String.format("Source file '%s' does not exist", resolvedSource));
}
return processListingFile(path, destination);
}
}

return ReactiveFileUtils.createDirectories(destination)
.flatMapMany(ignored -> {
if (Uris.isUri(resolvedSource)) {
return processRemoteSource(resolvedSource, destination);
} else {
final Path path = Paths.get(resolvedSource);
if (!Files.exists(path)) {
return Mono.error(new GenericException(String.format("Source file '%s' does not exist", resolvedSource)));
}

if (Files.isDirectory(path)) {
return processDirectory(path, destination);
} else {
return processFile(path, destination);
}
}
});
}

private Flux<Path> processListingFile(Path listingFile) {
private Flux<Path> processListingFile(Path listingFile, Path destination) {
return Mono.just(listingFile)
.publishOn(Schedulers.boundedElastic())
.flatMapMany(path -> {
Expand All @@ -141,22 +173,23 @@ private Flux<Path> processListingFile(Path listingFile) {
.filter(this::isListingLine)
.flatMap(src -> processSource(src,
// avoid recursive file-listing processing
false));
false,
destination));
} catch (IOException e) {
return Mono.error(new GenericException("Failed to read file listing from " + path));
}
});
}

private Mono<Path> processFile(Path source) {
private Mono<Path> processFile(Path source, Path destination) {

return Mono.just(source)
.publishOn(Schedulers.boundedElastic())
.map(path -> processFileImmediate(source, dest));
.map(path -> processFileImmediate(source, destination));
}

/**
* Non-mono version of {@link #processFile(Path)}
* Non-mono version of {@link #processFile(Path, Path)}
*
* @param scopedDest allows for sub-directory destinations
*/
Expand Down Expand Up @@ -203,7 +236,7 @@ private Path processFileImmediate(Path source, Path scopedDest) {
return destFile;
}

private Flux<Path> processDirectory(Path srcDir) {
private Flux<Path> processDirectory(Path srcDir, Path destination) {
return Mono.just(srcDir)
.publishOn(Schedulers.boundedElastic())
.flatMapMany(path -> {
Expand All @@ -220,7 +253,7 @@ private Flux<Path> processDirectory(Path srcDir) {
try (DirectoryStream<Path> files = Files.newDirectoryStream(srcDir, fileGlob)) {
for (final Path file : files) {
//noinspection BlockingMethodInNonBlockingContext because IntelliJ is confused
results.add(processFileImmediate(file, dest));
results.add(processFileImmediate(file, destination));
}
}
return Flux.fromIterable(results);
Expand All @@ -230,10 +263,10 @@ private Flux<Path> processDirectory(Path srcDir) {
});
}

private Mono<Path> processRemoteSource(String source) {
private Mono<Path> processRemoteSource(String source, Path destination) {
return Fetch.fetch(URI.create(source))
.userAgentCommand("mcopy")
.toDirectory(dest)
.toDirectory(destination)
.skipUpToDate(skipUpToDate)
.skipExisting(skipExisting)
.handleDownloaded((downloaded, uri, size) ->
Expand Down Expand Up @@ -263,7 +296,7 @@ private Mono<Path> processRemoteSource(String source) {
.checkpoint("Retrieving " + source, true);
}

private Flux<Path> processRemoteListingFile(String source) {
private Flux<Path> processRemoteListingFile(String source, Path destination) {
@SuppressWarnings("resource") // closed on terminate
SharedFetch sharedFetch = Fetch.sharedFetch("mcopy", SharedFetch.Options.builder().build());
return Mono.just(source)
Expand All @@ -273,7 +306,7 @@ private Flux<Path> processRemoteListingFile(String source) {
.flatMapMany(content -> Flux.just(content.split("\\r?\\n")))
.filter(this::isListingLine)
)
.flatMap(this::processRemoteSource)
.flatMap(url -> processSource(url, false, destination))
.doOnTerminate(sharedFetch::close)
.checkpoint("Processing remote listing at " + source, true);
}
Expand Down
Loading