pytmd download bug

[dbuscombe-usgs]

several users encountering this issue with pyfes download

it starts to download files, but then errors. I'm attempting to do this on a govt network, but I believe the problem manifests over home networks too.

I'll continue to investigate, It could be user error, aviso server issues, or related to SSL or other permissions ....

[2320sharon]

I believe the the cause of the error is in the ftp_download_file function that I have below. If the filename downloaded somehow doesn't include 'load_tide' or 'ocean_tide' in its name then the list of missing files is not created, thus when the line if len(missing_files) == 0: is run, it causes an error. Luckily the code can easily be refactored to fix this which I am happy to take care of.

Thanks for reporting this issue. :)

• The question is now file are some of the files not named 'load_tide' or 'ocean_tide'? What are they named instead?

def ftp_download_file(logger,ftp,remote_path,local_dir,tarmode,flatten,GZIP,MODE):
    """
    The function performs the following tasks:
    1. Retrieves the remote file paths and constructs an FTP URL, logging this information.
    2. If tar mode is specified: a. Reads binary data from the remote file and stores it in a bytesIO object. b. Opens the tar file and reads its content. c. Extracts all files in the tar file. d. Depending on the file extension and whether GZIP compression is enabled or not, the files are either compressed with gzip or simply copied to the local directory. e. Stores the last modified date of the remote file and applies it to the local file while retaining the local access time.
    3.If tar mode is not specified, the function copies readme and uncompressed files directly to the local directory, keeping the remote modification time and local access time.

    Args:
        logger (logging object): logger: A logging object used to log events and messages during the process
        ftp (_type_): A connection to an FTP server to download the files
        remote_path (_type_): The path where the files are located on the FTP server
        local_dir (str): The path where the files should be downloaded on the local machine
        tarmode (_type_): The mode in which the tar files should be processed, if applicable
        flatten (bool): A boolean indicating whether the downloaded files should be flattened or not
        GZIP (bool): A boolean indicating whether gzip compression should be applied to the downloaded files
        MODE (_type_): Unix file permissions for downloaded files

    """    
    remote_file = posixpath.join('auxiliary','tide_model',*remote_path)

    # check if local file exists
    local_file = os.path.join(local_dir,remote_path[-1])
    if os.path.exists(local_file):
        logger.info(f'File {local_file} already exists, skipping download.')
        return
    
    filename = os.path.basename(local_file)
    if 'ocean_tide' in filename:
        ocean_tide_dir = os.path.join(local_dir,'ocean_tide')
        missing_files = check_files(ocean_tide_dir,ocean_tide_files)
    if 'load_tide' in filename:  
        load_tide_dir = os.path.join(local_dir,'load_tide')
        missing_files = check_files(load_tide_dir,load_tide_files)
    
    if len(missing_files) == 0:
        print(f"All the required files have already been downloaded")
        return