Skip to content

Commit

Permalink
[DOCFIX] Add warning for distributedCp limitations
Browse files Browse the repository at this point in the history
There are known limitations in using the `fs distributedCp` command; explicitly call them out in the docs, ex. https://docs.alluxio.io/ee-da/user/stable/en/operation/User-CLI.html#distributedcp
			pr-link: #18608
			change-id: cid-be4807b887956808990a5094b4afee63780bfd84
  • Loading branch information
Xenorith authored and alluxio-bot committed May 15, 2024
1 parent 0546554 commit 8f490b0
Showing 1 changed file with 6 additions and 0 deletions.
6 changes: 6 additions & 0 deletions docs/en/operation/User-CLI.md
Original file line number Diff line number Diff line change
Expand Up @@ -847,6 +847,12 @@ Please wait for command submission to finish..
Submitted migrate job successfully, jobControlId = JOB_CONTROL_ID_2
```

Please note below are known limitations for the distributed copy command.
- Limited Scalability: No more than 1 million total number of files should be moved concurrently. Note that a copy job may stay active for a short period after the last file is copied.
- Manual Integrity Validation: Verification between source and destination files relies on the response code from the underlying data lake storage. In case the response code is unreliable, we recommend manual verification of source and destination checksums.
- Manual Cleanup: In certain failure scenarios, a user may need to manually remove partially written contents in destination directories and restart the failed jobs.
- Limited Observability: Status checks are limited to using the command line for each job individually.

### du

The `du` command outputs the total size and amount stored in Alluxio of files and folders.
Expand Down

0 comments on commit 8f490b0

Please sign in to comment.