copy_from_remote_to_remote question

[KimBenjaminTang]

I am new to AiiDA and installed version v0.12.0 and wanted to find out how I can use the copy_from_remote_to_remote method to copy a file from one remote computer to another.

I don't understand how exactly I can use it and how the parameters need to be defined.

From this here I know that the last two parameters remote source and destination need to be paths, but how exactly is transportsource and transportdestination defined?

It would be very helpful if there is a code example or other tutorial available and I'm sorry if it is something obvious but I struggle with understanding how to use it.

Thanks for helping!

[unkcpz]

Besides a paragraph for how to use this method (And it should be the method as a util aiida/transports/utils.py:copy_from_remote_to_remote), a test is good to have first and show how to use this function.

Moreover, I think it would be great to have a command-line interface to copy files from remote to remote. But in which verdi sub-group code? Is new verdi maintenance suitable for this @ramirezfranciscof ?

[Deepanshu7777777]

hey @KimBenjaminTang
Sure, I can help you with that! The copy_from_remote_to_remote method in AiiDA allows you to copy a file from a remote computer to another remote computer, by specifying the transportsource and transportdestination parameters.

Here is an example code snippet that shows how to use copy_from_remote_to_remote method in AiiDA:
from aiida import load_profile
load_profile()

from aiida.orm import Computer
from aiida.transports import TransportFactory

Define the source and destination computers

source_computer = Computer.get('source_computer')
dest_computer = Computer.get('dest_computer')

Define the source and destination transport objects

source_transport = source_computer.get_transport()
dest_transport = dest_computer.get_transport()

Define the source and destination paths

source_path = '/path/to/source/file.txt'
dest_path = '/path/to/destination/file.txt'

Copy the file from the source to the destination

source_transport.copy_from_remote_to_remote(
remote_path=source_path,
remote_computer=source_computer,
transportdestination=dest_transport,
destination=dest_path,
)

[VinaySingh561]

Hi @KimBenjaminTang
To copy a file between two remote computers using copy_from_remote_to_remote in AiiDA, follow these steps:

Step-by-Step Explanation

1. Load the Computers: Retrieve the Computer objects for the source and destination from the AiiDA database.
2. Set Up Transports: Create transport instances for each computer using their configured transport type (e.g., SSH).
3. Authenticate Transports: Open the transports using the user's authentication details stored in AiiDA.
4. Copy the File: Use copy_from_remote_to_remote with the open transports and the respective remote paths.
5. Close Transports: Ensure transports are closed after the operation.

Example Code

from aiida import orm
from aiida.engine.utils import copy_from_remote_to_remote
from aiida.transports import TransportFactory

# Load source and destination computers
source_computer = orm.load_computer('source_computer_name')
destination_computer = orm.load_computer('destination_computer_name')

# Get default user (or specify another user)
user = orm.User.objects.get_default()

# Retrieve authentication info for the user
source_authinfo = source_computer.get_authinfo(user)
destination_authinfo = destination_computer.get_authinfo(user)

# Create transport objects
Transport = TransportFactory
source_transport = Transport(source_computer.transport_type)
destination_transport = Transport(destination_computer.transport_type)

# Open transports (connect to remotes)
source_transport.open(source_authinfo)
destination_transport.open(destination_authinfo)

# Define remote paths
source_path = '/path/source_file.txt'
destination_path = '/path/destination_file.txt'

# Copy the file
copy_from_remote_to_remote(
    source_transport,
    destination_transport,
    source_path,
    destination_path
)

# Close transports
source_transport.close()
destination_transport.close()

Key Notes:

• Transports: The source_transport and destination_transport handle connections to the respective remote computers. They use the transport type (e.g., ssh) defined in each Computer's configuration.
• Paths: Ensure the source path exists on the source computer and the destination path is writable. Create directories on the destination if needed using destination_transport.makedirs().
• Authentication: AiiDA automatically uses the stored SSH keys or passwords configured for each computer, so no manual credential handling is needed.
• Error Handling: Wrap the code in try-except blocks to handle exceptions like connection errors or missing files.

Using Context Managers (Recommended)

To ensure transports are properly closed, use context managers:

with source_transport as s_t, destination_transport as d_t:
    copy_from_remote_to_remote(s_t, d_t, source_path, destination_path)

This automatically opens and closes the transports, reducing the risk of leaving connections open.

By following this approach, you can securely copy files between remote computers within your AiiDA workflows.