Verseles\Flyclone

PHP wrapper for rclone

Flyclone provides an intuitive, object-oriented interface for interacting with rclone, the powerful command-line program for managing files on cloud storage.

Key Features

• Broad Provider Support: Works with numerous storage backends supported by rclone (see below).
• Fluent API: Simplifies rclone command execution.
• Progress Reporting: Built-in support for tracking transfer progress.
• Process Management: Handles rclone process execution, timeouts, and errors.
• Easy Configuration: Configure providers and rclone flags directly in PHP.

Supported Providers

Flyclone supports a wide array of rclone providers, including:

• Local filesystem (local)
• Amazon S3 & S3-compatible (e.g., MinIO) (s3)
• SFTP (sftp)
• FTP (ftp)
• Dropbox (dropbox)
• Google Drive (drive)
• Mega (mega)
• Backblaze B2 (b2)
• ...and many others supported by rclone. New providers can often be used by leveraging the generic Provider class or by adding specific classes via PR.

Installation

composer require verseles/flyclone

Requires PHP >= 8.4.

Usage

Configuration Basics

1. Provider Setup: Each storage backend (local disk, S3 bucket, SFTP server, etc.) is represented by a Provider class. You'll instantiate a provider with a unique nickname and its rclone configuration parameters.

2. Obscuring Secrets: Rclone (and therefore Flyclone) often requires sensitive information like API keys or passwords. It's highly recommended to use rclone's obscure feature for passwords. Flyclone provides a helper for this:

use Verseles\Flyclone\Rclone;

$obscuredPassword = Rclone::obscure('your-sftp-password');
// This $obscuredPassword can then be used in the provider configuration.

3. Rclone Binary Path (Optional): Flyclone attempts to locate the rclone binary automatically. If it's installed in a non-standard location, you can specify the path:

Rclone::setBIN('/path/to/your/rclone_binary');

Instantiating Rclone

You create an Rclone instance with one or two providers:

• One Provider: For operations on a single remote (e.g., listing files, creating directories, moving files within the same remote).
• Two Providers: For operations between two different remotes (e.g., copying from local to S3, syncing SFTP to Dropbox).

use Verseles\Flyclone\Rclone;
use Verseles\Flyclone\Providers\LocalProvider;
use Verseles\Flyclone\Providers\S3Provider;

// Operations on a single local disk
$localDisk = new LocalProvider('myLocalDisk');
$rcloneLocal = new Rclone($localDisk);

// Operations between local disk and an S3 bucket
$s3Bucket = new S3Provider('myS3Remote', [
    'region' => 'us-east-1',
    'access_key_id' => 'YOUR_ACCESS_KEY',
    'secret_access_key' => 'YOUR_SECRET_KEY',
    // 'endpoint' => 'https://your.minio.server' // For S3-compatible like MinIO
]);
$rcloneS3Transfer = new Rclone($localDisk, $s3Bucket); // $localDisk is source, $s3Bucket is destination

Common Operations

List files (ls)

use Verseles\Flyclone\Rclone;
use Verseles\Flyclone\Providers\LocalProvider;
use Verseles\Flyclone\Providers\S3Provider;

// Example 1: List local files
$local = new LocalProvider('homeDir');
$rclone = new Rclone($local);
$files = $rclone->ls('/home/user/documents'); // Path on the 'homeDir' remote
/*
$files will be an array of objects, e.g.:
[
    (object) [
        "Path" => "report.docx",
        "Name" => "report.docx",
        "Size" => 12345,
        "MimeType" => "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
        "ModTime" => 1678886400, // Unix timestamp
        "IsDir" => false
    ],
    (object) [
        "Path" => "archive",
        "Name" => "archive",
        "Size" => -1, // Typically -1 for directories with rclone lsjson
        "MimeType" => "inode/directory",
        "ModTime" => 1678886500,
        "IsDir" => true
    ]
]
*/
var_dump($files);

// Example 2: List files from an S3 bucket
$s3 = new S3Provider('myS3', [ /* S3 config */ ]);
$rcloneS3 = new Rclone($s3);
$s3Files = $rcloneS3->ls('my-bucket-name/path/to/folder');
var_dump($s3Files);

Create a directory (mkdir)

use Verseles\Flyclone\Rclone;
use Verseles\Flyclone\Providers\SFtpProvider;

$sftp = new SFtpProvider('mySFTP', [
    'host' => 'sftp.example.com',
    'user' => 'user',
    'pass' => Rclone::obscure('password')
]);
$rclone = new Rclone($sftp);

$rclone->mkdir('/remote/path/new_directory'); // Creates 'new_directory' on SFTP server

Copy files/directories (copy, copyto)

use Verseles\Flyclone\Rclone;
use Verseles\Flyclone\Providers\LocalProvider;
use Verseles\Flyclone\Providers\S3Provider;

$local = new LocalProvider('myDisk');
$s3 = new S3Provider('myS3', [ /* S3 config */ ]);
$rclone = new Rclone($local, $s3); // local is source, S3 is destination

// Copy a local directory to S3
// Copies contents of /local/data to s3://my-bucket/backups/data
$rclone->copy('/local/data', 'my-bucket/backups/data');

// Copy a single local file to S3 with a specific name
// Copies /local/file.txt to s3://my-bucket/target/renamed.txt
$rclone->copyto('/local/file.txt', 'my-bucket/target/renamed.txt');

Move files/directories (move, moveto)

use Verseles\Flyclone\Rclone;
use Verseles\Flyclone\Providers\LocalProvider;

$localDisk = new LocalProvider('myDisk');
$rclone = new Rclone($localDisk); // Operations on the same local disk

// Move a file to another location on the same disk (effectively renaming)
$rclone->moveto('/old/path/file.txt', '/new/path/renamed_file.txt');

// To move between different remotes:
$sftp = new SFtpProvider('mySFTP', [ /* config */ ]);
$rcloneTransfer = new Rclone($localDisk, $sftp); // Local to SFTP
$rcloneTransfer->move('/local/source_folder', '/remote_sftp/destination_folder');

Sync directories (sync)

use Verseles\Flyclone\Rclone;
use Verseles\Flyclone\Providers\LocalProvider;
use Verseles\Flyclone\Providers\SFtpProvider;

$local = new LocalProvider('myDocs');
$sftpBackup = new SFtpProvider('sftpBackup', [ /* config */ ]);
$rclone = new Rclone($local, $sftpBackup); // Sync from local to SFTP

// Make SFTP /backup/documents identical to local /user/documents
// Only transfers changed files, deletes files on SFTP not present locally.
$rclone->sync('/user/documents', '/backup/documents');

Delete files/directories (delete, deletefile, purge, rmdir)

use Verseles\Flyclone\Rclone;
use Verseles\Flyclone\Providers\S3Provider;

$s3 = new S3Provider('myS3', [ /* config */ ]);
$rclone = new Rclone($s3);

// Delete a single file
$rclone->deletefile('my-bucket/path/to/file.txt');

// Delete all *.log files in a directory (respects filters)
$rclone->delete('my-bucket/logs/', ['include' => '*.log']);

// Remove an empty directory
$rclone->rmdir('my-bucket/empty_folder');

// Remove a directory and all its contents (does NOT respect filters)
$rclone->purge('my-bucket/old_stuff_to_delete_completely');

Check existence (is_file, is_dir)

use Verseles\Flyclone\Rclone;
use Verseles\Flyclone\Providers\LocalProvider;

$local = new LocalProvider('myDisk');
$rclone = new Rclone($local);

$fileExists = $rclone->is_file('/path/to/some/file.txt');
if ($fileExists->exists) {
    echo "File exists. Size: " . $fileExists->details->Size;
}

$dirExists = $rclone->is_dir('/path/to/some/directory');
if ($dirExists->exists) {
    echo "Directory exists.";
}
/*
 $fileExists / $dirExists object structure:
 (object) [
     'exists' => true, // or false
     'details' => (object) [...], // rclone lsjson item details if exists, or empty array []
     'error' => '' // or Exception object if ls failed
 ]
*/

Read file content (cat)

use Verseles\Flyclone\Rclone;
use Verseles\Flyclone\Providers\LocalProvider;

$local = new LocalProvider('myDisk');
$rclone = new Rclone($local);

$content = $rclone->cat('/path/to/config.ini');
echo $content;

Write content to a file (rcat)

use Verseles\Flyclone\Rclone;
use Verseles\Flyclone\Providers\SFtpProvider;

$sftp = new SFtpProvider('mySFTP', [ /* config */ ]);
$rclone = new Rclone($sftp);

$newContent = "Hello from Flyclone!";
$rclone->rcat('/remote/path/newfile.txt', $newContent);

Get size of files/directories (size)

use Verseles\Flyclone\Rclone;
use Verseles\Flyclone\Providers\S3Provider;

$s3 = new S3Provider('myS3', [ /* config */ ]);
$rclone = new Rclone($s3);

$sizeInfo = $rclone->size('my-bucket/some_folder');
/*
$sizeInfo will be an object, e.g.:
(object) [
    "count" => 150,
    "bytes" => 1073741824 // 1 GiB
]
*/
echo "Total files: {$sizeInfo->count}, Total bytes: {$sizeInfo->bytes}";

Upload a local file (upload_file)

use Verseles\Flyclone\Rclone;
use Verseles\Flyclone\Providers\S3Provider;

$s3 = new S3Provider('myS3', [ /* S3 config */ ]);
$rclone = new Rclone($s3); // $s3 is the destination for uploads

// Uploads /tmp/local_file.zip to s3://my-bucket/uploads/local_file.zip
// The local file /tmp/local_file.zip is removed after successful upload (uses rclone moveto).
$rclone->upload_file('/tmp/local_file.zip', 'my-bucket/uploads/local_file.zip');

Download a remote file (download_to_local)

use Verseles\Flyclone\Rclone;
use Verseles\Flyclone\Providers\SFtpProvider;

$sftp = new SFtpProvider('mySFTP', [ /* config */ ]);
$rclone = new Rclone($sftp); // $sftp is the source for downloads

// Download from SFTP to a specific local path
$localPath = $rclone->download_to_local('/remote/path/on_sftp/document.pdf', '/home/user/downloads/document.pdf');
if ($localPath) {
    echo "Downloaded to: " . $localPath;
}

// Download to a temporary directory (filename preserved)
$tempPath = $rclone->download_to_local('/remote/path/on_sftp/image.jpg');
if ($tempPath) {
    echo "Downloaded to temporary location: " . $tempPath;
    // Remember to unlink($tempPath) and rmdir(dirname($tempPath)) when done if temporary.
}

Copy with progress reporting

use Verseles\Flyclone\Rclone;
use Verseles\Flyclone\Providers\LocalProvider;
use Verseles\Flyclone\Providers\DropboxProvider; // Example with Dropbox

$local = new LocalProvider('myLocal');
$dropbox = new DropboxProvider('myDropbox', [
    'client_id'     => 'YOUR_DROPBOX_CLIENT_ID',
    'client_secret' => 'YOUR_DROPBOX_CLIENT_SECRET',
    'token'         => 'YOUR_DROPBOX_TOKEN', // Get this via rclone config
]);

$rclone = new Rclone($local, $dropbox);

$sourceFile = '/path/to/large_local_file.zip';
$destinationPath = '/dropbox_folder/'; // Directory on Dropbox

$rclone->copy($sourceFile, $destinationPath, [], static function ($type, $buffer) use ($rclone) {
    // $type is \Symfony\Component\Process\Process::OUT or \Symfony\Component\Process\Process::ERR
    // $buffer contains the raw rclone progress line
    if ($type === \Symfony\Component\Process\Process::OUT && !empty(trim($buffer))) {
        $progress = $rclone->getProgress(); // Get structured progress object
        /*
        $progress might look like:
        (object) [
            'raw' => '1.234 GiB / 2.000 GiB, 61%, 12.345 MiB/s, ETA 1m2s (xfr#1/1)',
            'dataSent' => '1.234 GiB',
            'dataTotal' => '2.000 GiB',
            'sent' => 61, // Percentage
            'speed' => '12.345 MiB/s',
            'eta' => '1m2s',
            'xfr' => '1/1' // Files being transferred / total files in this batch
        ]
        */
        printf(
            "\rProgress: %d%% (%s / %s) at %s, ETA: %s, Files: %s",
            $progress->sent,
            $progress->dataSent,
            $progress->dataTotal,
            $progress->speed,
            $progress->eta,
            $progress->xfr
        );
    }
});
echo "\nCopy complete!\n";

Advanced Usage & Tips

• Rclone Documentation: Always refer to the official rclone documentation for detailed information on commands and flags. This library is a wrapper, so understanding rclone itself is beneficial.
• Flags: Any rclone flag (e.g., --retries, --max-depth) can be passed as the last array argument to most Flyclone methods. Convert flags like --some-flag value to ['some-flag' => 'value'] or --boolean-flag to ['boolean-flag' => true].

  $rclone->copy('/src', '/dest', ['retries' => 5, 'max-depth' => 3, 'dry-run' => true]);

• Single Provider Operations: If you instantiate Rclone with only one provider, operations like copy or move will assume the source and destination are on that same provider (e.g., moving files within the same S3 bucket).
• Global Rclone Settings:
• Rclone::setFlags(['checksum' => true, 'verbose' => true]): Set global flags for all subsequent rclone commands.
• Rclone::setEnvs(['RCLONE_BUFFER_SIZE' => '64M']): Set environment variables for rclone (these are usually prefixed with RCLONE_ automatically if not already).
• Rclone::setTimeout(300): Set the maximum execution time for rclone processes (seconds).
• Rclone::setIdleTimeout(120): Set the idle timeout for rclone processes (seconds).
• Error Handling: Flyclone throws specific exceptions based on rclone's exit codes (e.g., FileNotFoundException, DirectoryNotFoundException, TemporaryErrorException). Catch these for robust error management.

To-do

• Add progress support
• Add timeout support
• Add more commands
• Add tests
  • Use docker and docker compose for tests
• Send meta details like file id in some storage system like google drive (e.g. for lsjson output).

Testing

Install Docker and Docker Compose, then run:

cp .env.example .env # Fill in any necessary credentials if you want to test against real cloud providers
make test-offline    # Runs tests against local, SFTP (Dockerized), S3/MinIO (Dockerized)
# or simply:
make                 # Default goal runs 'test-offline'

There are other test targets in the makefile (e.g., test_dropbox, test_gdrive), but they require you to fill the .env file with actual credentials for those services.

Contribution

You know the drill: Fork, branch, code, test, PR! Contributions are welcome.

License

Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International