Secure FTP Transfer Automation Pipeline

📖 Overview

This project is an automated Python pipeline designed for the reliable and secure transfer of generic data extracts.

It executes a decoupled, two-step transfer process:

1. Extraction: Connects to a source FTP server and downloads the latest data extracts to a local staging environment.
2. Delivery: Connects to a destination FTP server and uploads the staged files for downstream processing.

This decoupled architecture ensures that if the destination server is down, data is still safely extracted and staged locally, preventing data loss or pipeline corruption.

---

🏛️ System Architecture & Workflow

1. Initialization: The script securely loads authentication credentials from a hidden .env file.
2. Staging Preparation: Verifies or creates a local input directory.
3. Source Connection: Authenticates with the Source FTP, lists available files, and downloads them in binary mode (RETR) to the local input folder.
4. Destination Connection: Authenticates with the Destination FTP, verifies the target directory exists (creates it if missing), and uploads the staged files in binary mode (STOR).
5. Auditing: All steps, successes, and failures are written simultaneously to the console and a persistent local log file.

Expected Directory Structure

Secure_FTP_Transfer/
│
├── secure_ftp_transfer.py   # The main execution script
├── test_server.py           # (Optional) Local sandbox testing script
├── .env                     # Hidden configuration file (DO NOT COMMIT)
├── ftp_transfer.log         # Auto-generated audit log
└── input/                   # Auto-generated local staging folder

---

⚙️ Setup & Configuration

1. Prerequisites

• OS: Windows 10 / Windows Server
• Python: Python 3.7 or higher
• Network: Outbound port 21 (FTP) access to source and destination servers.

2. Install Dependencies

Open Command Prompt and install the required modules:

pip install python-dotenv pyftpdlib pyopenssl

3. Secure Credential Configuration

To comply with security standards, no credentials are hardcoded in the script. You must create a configuration file named exactly .env in the same folder as your script.

Create the .env file and populate it with the following keys:

# ==========================================
# FTP Configuration Environment Variables
# ==========================================

# Source FTP (Where files are pulled FROM)
SRC_FTP_HOST=ftp.source-domain.com
SRC_FTP_USER=your_source_username
SRC_FTP_PASS=your_source_secure_password
SRC_FTP_DIR=/path/to/remote/source

# Destination FTP (Where files are pushed TO)
DEST_FTP_HOST=ftp.destination-domain.com
DEST_FTP_USER=your_dest_username
DEST_FTP_PASS=your_dest_secure_password
DEST_FTP_DIR=/output

# Protocol Security (FTPS vs FTP)
USE_SSL=True

Note: Ensure .env is added to your .gitignore file to prevent accidental credential leaks into source control.

---

🚀 Deployment (Long-Living Process)

The pipeline is designed to run continuously in the background and will execute transfers automatically every hour without needing external schedulers.

To deploy this script to a corporate environment:

Step 1: Configure the Interval

In your .env file, ensure you have set the wait interval in seconds.

# Defaults to 1 hour (3600 seconds)
RUN_INTERVAL_SECONDS=3600

Step 2: Run as a Background Service

Option A: Command Prompt (pm2 / nohup equivalent) If you are running this on a Windows Server, you can use built-in Windows utilities like pythonw.exe (which runs python without a console window) or a process manager like PM2 to ensure the script stays alive:

pythonw secure_ftp_transfer.py

(You can also use NSSM - the Non-Sucking Service Manager - to easily map this script as a formal Windows Service that auto-starts on boot).

Option B: Linux / WSL

nohup python3 secure_ftp_transfer.py > /dev/null 2>&1 &

---

🧪 Local Sandbox Testing

Before deploying to production, you can test the pipeline locally without touching external networks using the provided test_server.py utility.

1. Create a folder named FTP_Test in your project root. Inside it, create a source folder (containing a dummy .csv file) and an empty destination folder.
2. To test FTPS (TLS), generate a self-signed certificate:

   openssl req -x509 -newkey rsa:2048 -nodes -keyout keycert.pem -out keycert.pem -days 365 -subj "/CN=127.0.0.1"

3. Run the test server:

   python test_server.py

   (This spins up a local FTPS mock server on 127.0.0.1 port 2121 with user testuser / testpass).
4. Update your .env file to point SRC_FTP_HOST and DEST_FTP_HOST to 127.0.0.1 and ensure USE_SSL=True.
5. Open a new terminal and run the main script:

   python secure_ftp_transfer.py

6. Verify the dummy file was successfully moved from the source folder, through your local input folder, into the destination folder safely over TLS.

---

📋 Auditing & Troubleshooting

For compliance and operational triage, the script utilizes Python's built-in logging module.

• Log Location: ftp_transfer.log (located in the script's root directory).
• Audit Trail: The log records execution timestamps, successful authentications, file-by-file transfer statuses, and directory creation events.
• Troubleshooting:
  • Missing Files: Check the log to see if the script successfully authenticated but found 0 files at SRC_FTP_DIR.
  • Authentication Errors: Check the log for 530 Login incorrect errors, indicating a password change or expired account in the .env file.
  • Connection Timeouts: Indicates firewall blocks or incorrect hostnames. Verify outbound Port 21 access.