ChangeLog.txt

======================================================================================================
> WiiUFtpServer
======================================================================================================
Based on ftpiiu but with the following issues fixed :

    connections failures and crashs
    file's dates (timestamps)
    file injection (add rights on files uploaded)
    remove the one slot limitation on upload and unlock up to 8 simultaneous transfers (download/upload)
    much more faster than the original

WiiuFtpServer comes also with some extra features :

    support Wiiu PRO and Wiimote controllers
    you can choose to disable or enable the power saving feature
    enable / disable VERBOSE mode on server side
    mount NAND paths only if you ask for it

By default, NAND paths are not mounted.

NOTES :


    No user/password requiered and only one client is allowed

    The server does not implement the MTDM function (and so does not preserves files timestamps) but now displays the correct dates :
    
    Recommended FTP client settings :
        90 sec for timeout which ensure to not timeout on transfer
        99 or unlimited retry number
        0 sec between retries
        Auto : ASCII/bin
        auto : IPV4 / V6
        allow retry in active mode (full active mode is not working)

    Under windows, enable long filenames support with setting to 1 the value of HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem\LongPathsEnabled because some files created by the Wii-U have a name with more than 128 characters (up to 170) and so the limit of 255 characters for the entire path can easily be reached.

    I opened a ticket for FileZilla client that failed to transfer such files (https://trac.filezilla-project.org/ticket/12675#ticket)
    
======================================================================================================
BUILD : (i used Windows)

To build from scratch :

    install devkitPro (in DEVKITPRO_PATH)
    
Launch "msys2\msys2_shell.bat"

    export DEVKITPRO=$DEVKITPRO_PATH

    cd to WiiUFtpServer folder

    ./build.sh

It creates a HBL App under _sdCard\wiiu\apps\WiiUFtpServer

Then just copy the _sdCard folder content to your SD card.

======================================================================================================
HISTORY : 

2021/04/05 V1-0 : 
    > Increase bandwith by 30%
    > Set files timestamp to OStime instead of J1970
  
2021/04/22 V1-1 : 
    > Change rights on the files received : fix files injection (saves for WWD for example)
    > Increase bandwith by 50%

2021/04/22 V1-2 : 
    > Try to fix timestamps with a date offset
      time_t seems to be a long double timestamp with micro-seconds based on epoch 1980
      The following code return "floating-point type"
        if ((time_t)-1 > 0) {
            // time_t is an unsigned type
        }
        else if ((time_t)1 / 2 > 0) {
            // time_t is a signed integer type
        }
        else {
            // time_t is a floating-point type (I've never seen this)
        }                 
    > set version and date in meta.xml when building

2021/05/01 V2 : 
    > Fix channel creation (WUP package)
    > build.sh patch version and date in all xml files
    > remove debug code in main.c

2021/05/06 V2-1 : 
    > chmod also folder created by the client
    > [Request #1] Support for other controllers than gamepad
    
2021/05/09 V2-2 : 
    > fix regression : can't exit HBL version
      HBL : gamePad, wiiMote +? : OK
      HBC : only gamePad

2021/05/09 V3 : 
    > fix timestamps on files and folders

2021/05/18 V3-1 : 
    > always retry on transfert error
    > +15% on network speed by setting a max priority to the thread and 
      MAX_NET_BUFFER_SIZE to 512k

2021/05/28 V3-2 : Enhance display log on server side, multithreading just for the fun... (bottleneck is on network socket not on CPU), 
                  a real progress should be a SSL server using lib curl (should be more efficient than using select() for FTP sockets as 
                  the code is already asynchronous I/O)
    > remove unecessary characters from the logged line (shorter the line)
    > use default thread on CPU2 for the ftp serveur (priority 1)
    > set network connections on CPU0 (priority 0)
    > main process is launched (as usual) on the default thread of CPU1
    
2021/06/01 V4-0 :    
    > fix ip adress not shown

2021/06/07 V4-1 :    
    > feedback on settings
    > set max concurents connections to 32
    > fix bad version number shown in 4.0
    
2021/06/10 V4-2 :    
    > create a gitHub branch for the HBL version based on first version of dynamic_libs that gives a greater socket bandwith 
      almost 30% more than using the WUT API.
      This version will replace the HBL version in the future release (channel version is still based on libWUT)

2021/07/23 V4-3 :    
    > put the 2 codes in the same project :
        - version using dynamic libs for HBL (best bandwith ~40% faster than WUT version)
        - building channel using the WUT version
    > rename WUP package name to WiiUFtpServer

2021/08/05 V4-4 :    
    > fix regression caused by setvbuf (saves was corrupted because of unexpected behavior due to freeing buffer 
      before fclose on file)
    > disable Nagle (socket optimisation)

2021/08/07 V4-5 :    
    > increase bandwith by 240% by setting buffer optimization specifically to RCV/SND file sockets

2021/08/08 V5 :    
    > get extra KB/s by lowering more default socket buffers
    > finalize and merge dynlib / wut codes (WUT version gain 20% of bandwith)
    > fix issue when disconnecting/stop server (dynlibs)

2021/08/12 V5-1 :    
    > now allow simultaneous transfers also for upload (client -> server)
    > restore FTP thread (fixing v4-5 issue)
    > fix to 1 client max with 6 connections max
    > add a socket optimization : Suppress delayed ACKs
    > bring the channel version transferts speeds close to the HBL one

2021/09/01 V5-2 :    
    > mount NAND paths ONLY if asked (by pressing B) : a very good idea from Masana@GBATemp, thanks!!!
    > if auto shudown feature is enabled, pressing DOWN toggle the feature OFF/ON
    > add a verbose mode for server log (disable by default) that can be activated by pressing UP
    > longer value for internal timeout to avoid data transfer deconnexions
    > only 1 simultaneous transfers for up/download (fix deconnexion issues)
    > WUT is now also used for HBL version (since now it performs like dynamic libs one)
    > add some socket optimizations : 
        - Leave received OOB data in line (SO_OOBINLINE)
        - Disable slow start feature (SO_NOSLOWSTART)
        - Socket memory optimization (thanks to V10lator@GBATemp for your support!!!) 
     > now channel version exit to Wii-u Menu 

2021/09/02 V6 :    
    > fix a display issue when exit (missing a draw call)
    > + 10% speed (UL/DL) with using a double sized buffer (corresponding to the size
      really allocated by the system when using setsockopt on SO_RCVBUF & SO_SNDBUF)

    > replace WUT calls to MEMAllocFromDefaultHeapEx & MEMFreeToDefaultHeap in recv_to_file and send_from_file 
      that seems to be not stables (avoid -99 errors)
    > lower the memory consumption (lower buffers used in ftp.c and net.c)
    > enhance robustness by removing too agressive socket optimizations

2021/10/12 V6-1 :    
    > Wii-U PRO controller and WiiMote support. Thanks godreborn@GBATemp for testing!
    > tweaks for maximum performance :
        +10% DL speed compare to V6
        +70% UL speed (UL is now faster than DL)    
    > enhance robustness (fix errors -99, -100, ENOMEM, ECONABORTED...). thanks Masana@GBATemp for testing!
    > add a NAND backup/restore process :
        - when you choose to mount NAND paths, if no backup is found in WiiUFtpServer app folder on the SDcard,
          create a partial (2MB) or a FULL system files NAND backup (500MB free on SD card are requiered) 
        - if you brick your Wii-U with NO NETWORK AVAILABLE (so you can't start FTP) : restore only the files 
          needed (partial backup ones) to get WiiUFtpServer working again (then unbrick as the usual way)
          
2021/10/13 V6-2 :    
    > Restore originals headers in source files
    > unlimit the incomming connexions (hard limited for UL and DL in net.c)

2021/12/08 V7-0 :    
    > dived further in the code and rework the original sources (ftpiiu) to :
        - complete the work left (TODO) that essentially fix the limitation to one upload slot (vrt_chdir failure in ftp_MKD)
        - now server allows 6 simultaneous transfers on both ways (UL/DL)
        - add a transfer file module (asynchronous calls safe) to handle files transfering
        - add IO optimization on file (using a user buffer, socket one already implemented)
        - transfer speeds now are over 3MB/s in both ways (UL/DL)
    > set rights to 644 (instead of 666)
    > externalize dependencies (libIOSUHAX from yaWut and WUT library)
    > add a 2 level log file saved on the SDcard (disable by default if you compile sources because it impacts server performance) 
    > add LICENSE.txt file (forgot on my first fork...)
    > upgrade to Wut beta12
    > fix wrong timeout value when mounting devices
    > fix socket memory optimization failure

2021/12/26 V7-1 :    
    > HOTFIX : fix issue#02 by disabling user buffer's file optimization on upload (it corrupts files)
    > remove the module transferedFiles : all is now done in ftp (more close to the original design)
    > compute approx transfer rate on server side
    > push to 8 simultaneous transfers
    
2021/12/28 V7-2 :    
    > fix issue#03 (WWHD saves injection)
    > reduce to 7 the number of simultaneous transfers to avoid network overflow
    > finals tweaks on user's buffer to get a little bit higher perfomances than with V7-0
    
2021/12/31 V7-3 :    
    > fix low average speed computation for single transfer : now only consider files that fill the entire buffer
    > add a global speed stats (min/mean/max)
    > use the max amount of shared socket memory
    > use a larger file buffer (to enhance speed for transfering files larger than 1MB)

2022/01/01 V8-0 :    
    > multi-Threaded version : one thread per connection
    > this reduce network overflow and so reset to 8 for the max number of connections
    > overall transfer speed gain, mostly for download (+30% speed in multi transfer compared to V7)

2022/01/28 V8-1 :    
    > rewrite send_from_file() and recv_to_file() for non blocking mode and chunk based transfer (per block) : => +40/50% UL speed
    > pre-allocate a transfer buffer per connection
        - this buffer is used by both mode (download/upload)
        - it result in a faster transfer rate and mostly faster open/close times for connections
    > cancelling a transfert now cancel the thread (you have to wait the completion in V8-0 until getting back the connection)
    > display connection concerned in console log and client one
    > display transfer rate calculate on server in client's log
    > spinLocks, loop optimizations and cleanup WUT use mistakes by V10Lator
    > thanks also to Masana, godreborn and TheLegendOfXela for their checks on release candidate versions
    
2022/02/21 V8-2 :    
    > add a CRC32 calculation on server side. The report is saved under "wiiu/apps/WiiUFtpServer/crc32_report.sfv" with
      a 2 level (.old to backup the previous session).
    > this report and the log file are now located in mlc or usb/usr/tmp
    > add a CRC32 verification tool (windows version, i'll release a linux's version later)
    > fix issue#10 (remove the resizing of internal file's buffer)
        - rare corruption on uploaded file (~ 1 file on 3000)
        - support cyberduck FTP client (enhance global FTP clients support)
    > change libiosuhax (YaWut) to official one (Wiiu-Env)
    > fix refresh display issue when sending msg from threads
    > add the possibility to toggle ON/OFF the auto-shutdown feature now during the FTP session
    > multi-transfers enhancement : 
        - optimize controller events monitoring (and so the main loop)
        - final tweaks on thread priorities to optimize the connections dispatching on the 3 cores
    > upload single transfer enhancement : nearly double the size of the transfer buffer for upload
      use a size of TRANSFER_BUFFER_SIZE + TRANSFER_CHUNK_SIZE instead of use 2 * TRANSFER_BUFFER_SIZE to handle 
      network_readChunk() overflow, knowing that TRANSFER_CHUNK_SIZE is the max number of bytes returned by recv()
    > lower the RAM consumption (now less than 800MB)    
    
2022/02/28 V9 :
    > change CRC format to handle space in path/file's name
    > after bunch of tests, CRC32 is now disabled by default but can be toggled at anytime during session using 'X' button
    > clever handling of sdCard copy of CRC report file : update the copy when only one connection is opened and if the
      number of files transferred has increased. Deactivate the copy process as soon as 2 connections are opened
    > embeded the CRC checker on a subfolder on the SDCard, the report is now saved in this subfolder
    > lower "stall time" due to changing the rights on file uploaded. Stall time = writing/flush the network
      buffer (transfer rate on upload does not take into account the writing to file time = that's why the
      transfer rate is so high in upload way). Use a system'scall instead of IOSUHAX_FSA_ChangeMode
      function (which seems to re-allocate the buffer for the operation???)
    > lower CWD errors numbers (it is normal to get some of them when folders do not exist)      
    > do not timeout when a connection was opened and "stall"       
    > final tweaks on thread priorities (multi-transfer enhancement)
    > revert log file location to SDCard (was a bad idea because in case of crash, the log on SDCard is outdated)
    > CrcChecker V2 in python (Linux/window) 30 times faster than V1-2 bat file
    > update CrcChecker V1-3 (bat file) to the new CRC32 report format

2022/03/15 V9-1 :
    > add symlinks resolution support for the ones found in updates and DLC folders (fix transfer errors on those files)
    > move CRC32 calculation after the analysis of the return code of fwrite() in net::recv_to_file
    > fix poor transfer speeds to SDCard with using libFat (it also fix app hang/cripple when activating the log file)
      HARD FIXED the max simultaneous transfers to SDCard to 4 (to avoid write errors)
    > fix issue#14 : Transfer large size file on SDcard, WIIUFTPSERVER disconnect? 
    > rollback to IOSUHAX_FSA_ChangeMode because rights was not correctly set using system call.
    > lower the preallocated transfer buffer to ~13MB 
      So the total RAM used is ~110MB of RAM used (when transferring 8 files)
      It lowers the ending closing connection time but also gives extra KB/s
    > add folder deletion support
    > fix a regression on renaming file process
    > crcChecker V2-1 fix "TypeError: string operation on non-string array" that could occurs when treating the report

2022/03/19 V9-2 :
    > remove DEBUG traces left in ftp.c and vrt.c
    > add a 10s timeout for the CRC32 Warning
    > final tweaks to optimize multi transfers (numbers of transfers launched simultaneously)
    > display current transfer speed stats when no transfer is active and when only a browse connection is opened
    > build.sh : fix build date not replaced in meta.xml 
    > add ./toWUP/createChannel.sh (Linux shell version) 
    > add a custom bootSound.btsnd to channel version (sample created from "ZOMBIE" by Fela Kuti)

2022/04/01 V9-3 :    
    > use libFat and libIOSUHAX versions from Crementif (thanks!!!) : better timestamps and symplinks support 
    > now symlinks are identified in FTP client browser
    > fix regression Cyberduck connection failure #15
    > fix FileZilla client fail to delete a folder #16 
    > fix WinScp fail to list root #17 
    > fix sd drive mount twice after the NAND files backup
    > add the ftp_RMD command (equivalent to ftp_DELE)
    > update meta.xml template
    > fix case for folders names in ./toWUP/createChannel.bat 
    > fix release date patching in meta.xml in build.sh 

2022/04/20 V10 :    
    > fix issue "Still rare CRC errors on upload (about 1 file on 9000 randomly) #19 ": 
      replace the preallocate buffer by an ontime allocation + use the transfer buffer as internal's file buffer (setvbuf)
    > multi transfers enhancement : double TRANSFER_BUFFER_SIZE for upload and lower the buffer used for download to the min
      +final tweaks on threads priorities to maximize the number of connections opened simultaneously on the 3 cores
    > update crcChecker.py to V2-2 :
        - fix answer "no" not taken into account when choosing to not use an report that already exists in the script's folder
        - add a try catch for permissions errors
        - fix failure when root = folder to check
    > move IOSUHAX_FSA_ChangeMode() call in the transfer thread (for uploads)
    > fix issue "Folder left after deletetion #18" 
    > fix "Crash when exiting if no connections were opened #20"

2022/05/18 V11 :    
    > fix issue " Still some CRC errors occur when uploading on SDCard #21 ": remove sevbuf use to fix the last CRC errors and limit 
      to only 1 transfer for sdCard operations (simultaneous transfers leads in CRC errors)  
    > multi transfer enhancement : reduce open/close connection time with 
        - using a static preallocated array of threads (before it was allocated statically when creating the connection) 
        - manually set cpu and priority of transfer threads
    > now use also a small buffer for UL operations (before setting manually CPU and priority, a large buffer was used to let other 
      connections start)
    > tweak the buffers sizes used for DL / UL
      Use 25% DL vs 75% UL of the reserved memory for buffer socket operations (instead of 50/50)
    > This last setting increase DL speeds obtain when multi-transferring. Using a small buffer for upload reduced the "speed displayed" 
      on UL operation (now take the fwrite time into account). 
      This version gives max speeds over 4M/s on both ways and treat a large number of files more quickly. 
      
2022/09/02 V12 :    
    > WUT developpement is interrupted (awaiting for a more stable version, tried 1.1.1 with no success), so no more channel version for 
      V12 and up.
    > it is a much more stable version that use dynamic libs (like the original software) and it fix : 
        - transfers number limitation on the SDCard, now 8 is allowed
        - CRC errors when transferring to the SDCard
      and i suppose : 
        - Disconnects to ftp server while copying a injected game #22 
        - Crash when trying to access a folder on SD #23 
        - unstable on tiramisu #24
        - It seems that using FileZilla client to delete files on sdcard caused it died. #26 
    > noticeable multi-transfers enhancement : 
        - now connections are re-used (no more alloc/free when client connects)
        - transfer buffer and thread are preallocated
        - extend the priority values used for transfer threads
    > fix : /storage_usb sometimes inaccessible if two USB devices are plugged in #25 
    > fix : renaming file is broken #27 
    > fix : rights on files uploaded not set properly
    > remove nand backup, log file and CRC calculation features
    
    Special thanks to Masana for testing.
        
2022/09/08 V12-1 :    
    > fix :  creating single letter folders Freezes the console #28 
    > optimize transfer buffer for DL (to ease connections start)