De-ID by file copy/rename?

[amg369]

Our MGH waveforms are in Bedmaster HD5 format, with the only PHI that I can identify being the filename itself, which is the PatientEncounterID. Our files are stored in a Windows shared network drive and I was debating the merits of copying the files to another shared location and renaming them there (since this the source is a common network drive and we don't have permissions to directly rename the files). However, this is extremely time consuming as some of these files are several GB. We're at this point considering creating a specific MGH only Azure cloud to move the files to, then rename when they are stored there.

In the meantime, does anyone have an efficient and relatively fast means of copying files in a Windows file structure or should we focus on bringing them up to an MGH Azure Cloud and then rename?

[del42]

@amg369 When your rename them, do you load them into memory? That takes time. There is way to not load them into memory and straight copy to another location with new name. Additionally, use your multiprocessing. I am sure you have several cores.

[del42]

Start with following... then customize it to your need.

import paramiko
from multiprocessing import Pool

def download_file_sftp(details):
    host, port, username, password, remote_filepath, local_filepath = details
    transport = paramiko.Transport((host, port))
    transport.connect(username=username, password=password)

    sftp = paramiko.SFTPClient.from_transport(transport)
    
    with sftp.open(remote_filepath, 'rb') as remote_file:
        with open(local_filepath, 'wb') as local_file:
            while True:
                data = remote_file.read(1024)  # Read in chunks of 1024 bytes
                if not data:
                    break
                local_file.write(data)

    sftp.close()
    transport.close()

def main():
    # List of files to download
    files_to_download = [
        ('your.sftpserver.com', 22, 'username', 'password', '/remote/path/file1.hdf5', '/local/path/file1.hdf5'),
        ('your.sftpserver.com', 22, 'username', 'password', '/remote/path/file2.hdf5', '/local/path/file2.hdf5'),
        # Add as many files as needed, assuming you have list of paths filtered
    ]

    # Number of processes
    pool_size = 4  # Adjust based on your pc, don't use all your cores, maybe 80% 
    with Pool(pool_size) as pool:
        pool.map(download_file_sftp, files_to_download)

if __name__ == "__main__":
    main()

[amg369]

The main trouble is that the files aren't stored in a remote server. Right now we only have a Windows shared network drive.

[del42]

So you need a code from local path to server? Can you run python script in that environment?

[amg369]

It's possible for me to run a python script and copy files from the shared network drive to another shared network drive - it's just extremely slow and tedious. It's mapped onto my local system, so I can use it like any other local drive.

[del42]

If it is archive drive, there is not much you can do. It is just hardware and network limitation. I don't know how your code is organized. But I would suggest, try not to traverse the whole file system to find something. Instead, copy all the paths first, locate all your files on your machine, then download your files with exact path.
Hope this helps.