lftp for Windows
featuring Windows task scheduler automation with pushover or pushbullet notifications.
What we were missing was a usable interface for lftp to bypass the steep learning curve.
What we needed was a way to use lftp on Windows provides a user friendly configuration whilst having no need to learn command line or install anything.
This project has provided a working and truly portable solution to that problem.
Using lftp on Windows is as simple as configuring a WinSCP connection profile via the template provided.
One login to rule them all!
Contents
Programs included and configured
Lftp mirror automation using task scheduler.
Introduction
The idea of this solution is to configure your servers as WinSCP sessions by editing the provided template session and saving it as a new modified session. Then via the WinSCP GUI you will be able to do these main things using one set of credentials:
1: Manage your server over sftp using WinSCP.
2: Easily connect using kitty to manage your server over ssh using the putty icon.
3: Download files and folders using specially crafted WinSCP custom commands that automatically configure and load lftp in ConEMU. You can download to a default or specified location in the background. When called, Lftp is run via an updated and modified version of the well known lftp sync script.
This makes this solution highly portable, highly useful and there is still a massive potential to extend and create functionality using a combination of Windows and Linux scripting.
One login to rule them all!
A single set of credentials will give yous ssh, sftp and lftp using either a password or putty key file.
Programs included and configured
All programs included are unmodified from their authors releases. They are only tweaked using available configuration files or settings. They retain whatever licensing they were released with.
WinSCP Portable - the main application that you will use
ConEmu x64 Portable - the windows terminal that runs the Cygwin binaries and interprets the bash scripts
Cygwin x64 binaries and a few other files for terminal functionality
Notepad ++ portable - Used with bat files to automatically load provided scripts for editing.
Kitty portable - loaded within the WinSCP GUI for SSH access to your configured servers
Kageant - required for key file + server authentication.
Usage
Download the latest release zip file and then extract it somewhere.
Notifications using Pushbullet or Pushover
Nice! If you don't have an account go make one here https://www.pushbullet.com/
The you need to visit this URL to generate an API key: https://www.pushbullet.com/#settings/account
Once you done that you need to run the Pushbullet Notifications.bat inside the lftp-windows directory and enter your key between the single quotes.
Note: If you want you can also change the notification message you receive here.
Once you have done that you will get a Pushbullet notification to your devices when either a mirror, pget or sync task it run and completes.
You can also use pushover.
Running WinSCP
Inside the lftp-windows directory you need to run the Double Click Me - WinSCP Startup.bat to load WinSCP.
Note: Use the template session provided as there are some per session settings unique to this template that work specifically with this set-up.
You just have to properly configure a new WinSCP session by editing the existing template session called sessioname and saving it as a new session. Move to the next section.
Password authentication (easiest solution with the least configuration)
Run the Double Click Me - WinSCP Startup.bat to load WinSCP without Kageant if you have no keys in the keys directory.
When you edit the template session in WinSCP, provide and save a password for authentication you will have a pretty straightforward experience.
Kitty and lftp will work with not further configuration required.
You can also configure a master password to protect your login:
You can move on the commands section once you are logged in to continue.
Private key authentication
Note: Please read this guide for using puttygen import or create key files.
Place your putty format ppk key file inside the keys directory.
kageant will load your Putty format key to use for authentication with WinSCP and Kitty.
ssh-pageant will use this key file to provide authentication to lftp.
Using your private key
This is fairly easy. All you need is to have your key file in the Putty format. You can create a new ppk format key using puttygen. It can also load and export an OpenSSH format key to the ppk format.
Note: Loading too many keys from the keys folder may result in authentication errors. I suggest you keep it simple and only have a few unique keys here.
1: You need to place the Putty format key in your Keys directory.
Keys/mykeyname.ppk
2: Run the Double Click Me - WinSCP Startup.bat to load all ppk keys in the Keys directory. If they are password protected you will need to enter this when prompted.
That is all. Your key will automatically be passed to lftp by the script.
Commands
Note: In WinSCP the local windows are on the left and the remote are on the right. Similar to Filezilla.
The hard coded download location is the /Downloads folder inside the lftp-winscp-portable directory.
There are 5 custom commands in total you need to understand.
Lftp Options opens a bash script to configure the transfer settings of the scripts. Use before you initiate a transfer to the settings to take effect.
Mirror and Pget are pretty simple:
Mirror - Will mirror your selected directory in the remote window to the hard coded downloads directory in the lftp-winscp-mirror.sh script.
Pget - Will pget your selected file in the remote window to the hard coded downloads directory in the lftp-winscp-pget.sh script.
The Mirror to dir and Pget to dir require correct usage.
You MUST select a folder in the local window (left bottom) for both commands. If you do not select a folder then it will not download to the right place. Most likely it will end up in the scripts folder. This means you cannot download to just a drive letter.
For example: You want to download to you Second drive F:\ so you must visit this drive and create a folder there. Then select this folder in the local windows of WinSCP. Now select the the folder or file you wish to download and use the correct command.
Mirror to Dir - This will mirror a directory to the selected destination in the local window of WinSCP - This command uses the lftp-winscp-mirror.sh script.
Pget to Dir - This will pget a file to the selected destination in the local window of WinSCP - This command uses the lftp-winscp-pget.sh script.
Scripts
There are three scripts provided. All three use the same hard coded lock file lftp-winscp.lock so as not to run if any other script is actioning a download. They will all check to see if lftp is actually running and clear dead lock files.
lftpsync.sh - A mirror script for use with ConEmu or as a scheduled task. The original idea of this project. Not used by WinSCP.
lftp-winscp-mirror.sh - A clone of the lftpsync.sh to be used and modified specifically with WinSCP.
lftp-winscp-pget.sh - A clone of the lftpsync.sh changed to use pget instead of mirror to be used and modified specifically with WinSCP.
lftp configuration
Most settings are hard coded in the scripts and it is recommend you edit the script to suit your specific needs.
Lftp can also use /system/.config/lftp/rc for global settings and /system.local/share/lftp for saving your bookmarks and logs (disabled by default in the global rc).
You can edit these files by using the help folder to run notepad ++.
help/start - notepad ++.bat
Help - how to use
There is a help folder with some tools to edit scripts,files and fix permissions.
remove usb.bat - Run this to close the main apps included. This is what the file contains.
taskkill /f /im "ssh-pageant.exe"
taskkill /f /im "kageant.exe"
taskkill /f /im "WinSCP.exe"
taskkill /f /im "bash.exe"
taskkill /f /im "ConEmu64.exe"
taskkill /f /im "notepad++.exe"
taskkill /f /im "kitty_portable.exe"start - ConEmu.bat - Load ConEmu to use the terminal.
start notepad++.bat - Will load the included notepad ++
Use these troubleshooting files with caution! They are not to be used casually.
troubleshooting - ownership - recycle bin.bat - Use only if you deleted transferred files that corrupted the recycle bin.
troubleshooting - ownership.bat - Use only if you are have file ownership issues with transferred files.
Lftp mirror automation using task scheduler
Please visit the automation directory and you will find the files below.
Step 1: You will need to run this 1 - edit lftpsync.sh.bat top edit the script with notepad ++
Step 2: Please hard code the required values into options 1 - 4. You don't have to change anything else, but may need to configure the optional settings if you have specific needs.
Step 3: Optional - Set a default download location using the format local_dir="/cygdrive/c/Downloads" where /cygdrive/DRIVELETTER/ is the path to the drive you want to use.
Step 4: Run the 2 - Create lftpsync task.bat to create a new basic task that runs the program lftpsync.bat at 20:00 every day.
Step 5: Optional - You can use the Optional - start taskscheduler.bat to tweak the ltfpsync task settings in the task scheduler.
Step 6: Optional - You can use the Optional - edit lftp options.bat to tweak the lftp settings or edit the script manually.
Step 7: Optional - You can use the Optional - test lftpsync task.bat to run the task now and check it works as intended.
Step 8: Optional - You can use the Optional - lftpsync settings tester.batto debug your connection settings. ConEmu will not run minimized or close upon completion.
Step 9: Optional - You can use the Optional - delete lftpsync task.bat delete lftpsync task from the task scheduler.
Congratulations, you are not using lftp on Windows on a schedule. Set it and forget it.
Updating
To update from one version to another all you really need to do is copy over the WinSCP.ini located in the \system\bin directory. This will migrate your logins from one version to another.
ConEmu: This program automatically checks for updates when running. It will prompt you.
WinSCP: You update this by downloading the portable binaries from the WinSCP website and placing the WinSCP.exe and WinSCP.com file in the system\bin directory.
Cygwin binaries: Such as lftp itself. To do this you need Cygwin x64 installed and you need to have also installed lftp, ssh-pageant and openssl. The run the update.bat in the \system\bin directory. This will copy the files from the Cygwin installation to your lftp-portable folder via this filelist.
Misc: There is an updater file in the \system\usr\ directory that updates some misc files from a Cygwin installation related to the functionality of ConEmu. If you have updated the Cygwin binaries as described above you should also do this periodically.
Known bugs
Possibly resolved - When mirror or pget to a custom directory lftp gets confused and creates an empty cygdrive folder shortly after the download starts. This removed by the scripts just before exiting. Never us rm on this /cygdrive as it will get interpreted as /cygdrive/c and try to wipe you C drive. Don't ask how I know this.
WinSCP is configured to use themtmp/ directory but it becomes relative to whatever local folder you are in when you try to edit a file so you might see a few tmp folders and winscp.rnd files in local directories you visited in WinSCP and performed a command that required some temporary files.
The other choice is to use the host machines temp directory as shown in this image.
