A lightweight Express.js API for comparing file sizes between rclone remotes and local directories, initially designed for use with Glance.
Report Bug
·
Request Feature
Table of Contents
- Compare sizes between rclone remotes and local directories
- Intelligent caching system to minimise repeated rclone operations
- Track size history for detecting when directories were last modified
- Automatic cache updates on configurable schedules
- Support for multiple remotes and directories
- Detailed logging system for monitoring operations
- RESTful API endpoints with comprehensive error handling
- Node.js (v14 or higher)
- npm or yarn
- rclone installed and configured with remotes
- Access to local directories for comparison
- Clone this repository:
git clone https://github.com/ssyyhhrr/rclone-reporter.git
cd rclone-reporter
- Install dependencies:
npm install
-
Create a
.envfile (see below) -
Start the server:
npm start
Create a .env file in the root directory with the following variables:
PORT=3000
LOG_DIR=./logs
GET /health
Returns the service status and cache information.
Example Response:
{
"status": "ok",
"timestamp": "2024-03-10T12:00:00.000Z",
"cacheStatus": {
"remote": {
"lastUpdated": "2024-03-10T11:00:00.000Z",
"remotesInCache": 3
},
"local": {
"lastUpdated": "2024-03-10T11:30:00.000Z",
"directoriesInCache": 5
}
}
}POST /api/compare
Body:
{
"remotePath": "myremote:path/to/directory",
"localPath": "/local/path/to/directory",
"forceDirect": false
}Example Response (Cache Hit):
{
"timestamp": "2024-03-10T12:00:00.000Z",
"remotePath": "myremote:path/to/directory",
"localPath": "/local/path/to/directory",
"lastModified": "2024-03-09T15:00:00.000Z",
"lastModifiedFormatted": "09/03/24 3PM",
"remote": {
"bytes": 5368709120,
"formatted": "5 GB",
"count": 150,
"cachedAt": "2024-03-10T11:00:00.000Z"
},
"local": {
"bytes": 5368709120,
"formatted": "5 GB",
"cachedAt": "2024-03-10T11:30:00.000Z"
},
"difference": {
"bytes": 0,
"formatted": "0 Bytes",
"direction": "equal"
},
"syncStatus": {
"percentageSynced": 100,
"isSynced": true
}
}Example Response (Cache Miss):
{
"status": "cache-miss",
"message": "Remote path \"myremote:path\" not found in cache. Use /api/cache/refresh to update the cache or set forceDirect=true in your request to fetch directly.",
"remotePath": "myremote:path",
"localPath": "/local/path",
"lastModified": "2024-03-09T15:00:00.000Z",
"lastModifiedFormatted": "09/03/24 3PM",
"local": {
"bytes": 1073741824,
"formatted": "1 GB",
"cachedAt": "2024-03-10T11:30:00.000Z"
},
"cacheStatus": {
"lastFullUpdate": "2024-03-10T11:00:00.000Z",
"updateInProgress": false,
"updateStartTime": null
}
}POST /api/cache/refresh
Triggers a manual cache update for all remotes and tracked local directories.
Example Response:
{
"status": "refresh-started",
"message": "Cache refresh has been initiated in the background for remote and local caches",
"remote": {
"updateStarted": true,
"startedAt": "2024-03-10T12:15:00.000Z",
"previousUpdate": "2024-03-10T00:00:00.000Z"
},
"local": {
"updateStarted": true,
"startedAt": "2024-03-10T12:15:00.000Z",
"previousUpdate": "2024-03-10T11:00:00.000Z",
"directoriesTracked": 5
}
}GET /api/cache/status
Returns detailed information about the current cache state.
Example Response:
{
"remote": {
"lastUpdated": "2024-03-10T11:00:00.000Z",
"updateInProgress": false,
"updateStartTime": null,
"remoteCount": 3,
"remotes": [
{
"path": "myremote:",
"size": "1.5 TB",
"bytes": 1649267441664,
"count": 50000,
"timestamp": "2024-03-10T11:00:00.000Z",
"calculationDuration": "45.2s"
}
]
},
"local": {
"lastUpdated": "2024-03-10T11:30:00.000Z",
"updateInProgress": false,
"updateStartTime": null,
"directoryCount": 5,
"directories": [
{
"path": "/mnt/data",
"size": "500 GB",
"bytes": 536870912000,
"timestamp": "2024-03-10T11:30:00.000Z",
"calculationDuration": "12.3s"
}
]
}
}# Compare remote and local directory
curl -X POST http://localhost:3000/api/compare \
-H "Content-Type: application/json" \
-d '{
"remotePath": "myremote:backup/photos",
"localPath": "/home/user/photos"
}'
# Refresh cache
curl -X POST http://localhost:3000/api/cache/refresh
# Check cache status
curl http://localhost:3000/api/cache/status// Compare directories
fetch('http://localhost:3000/api/compare', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
body: JSON.stringify({
remotePath: 'myremote:backup/photos',
localPath: '/home/user/photos'
}),
})
.then(response => response.json())
.then(data => console.log(data));
// Check cache status
fetch('http://localhost:3000/api/cache/status')
.then(response => response.json())
.then(data => console.log(data));- Ensure rclone is properly configured with all required remotes
- Ensure adequate permissions for accessing local directories
- Implement rate limiting if exposing the API publicly
- Consider adding authentication if the API will be accessed by multiple clients
If you have a suggestion that would make this project better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!
- Fork the Project
- Create your Feature Branch (
git checkout -b feature/AmazingFeature) - Commit your Changes (
git commit -m 'Add some AmazingFeature') - Push to the Branch (
git push origin feature/AmazingFeature) - Open a Pull Request
Distributed under the MIT License. See LICENSE.txt for more information.
- "Remote not found in cache": Run
/api/cache/refreshto update the cache, or wait for the next scheduled update. - Permission denied errors: Ensure the service has read access to all local directories being compared.
- rclone command not found: Make sure rclone is installed and available in the system PATH.
- High memory usage: The service caches all remote and local directory information. Reduce the number of tracked directories if memory is limited.
- Slow cache updates: Large remotes can take significant time to calculate. Consider increasing the update interval or using fewer remotes.
Rhys Bishop - https://sy.hr/ - mail@rhysbi.shop
Project Link: https://github.com/ssyyhhrr/rclone-reporter