Runtime error with OpenSSL 1.1.0f and mod_sftp (was ok with OpenSSL 1.0.1f)

[Vincent--]

What I Did

Trying to connect with a simple command line:
sftp -vvv -o PasswordAuthentication=yes -P 2222 example@127.0.0.1

Get disconnected on the client side:

# sftp -vv -o PasswordAuthentication=yes -P 2222 example@127.0.0.1
OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014
debug1: Reading configuration data /root/.ssh/config
debug3: ciphers ok: [aes256-ctr]
debug3: macs ok: [hmac-sha2-512]
debug3: kex names ok: [diffie-hellman-group14-sha1]
debug1: Reading configuration data /etc/ssh/ssh_config
debug3: ciphers ok: [chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr]
debug1: /etc/ssh/ssh_config line 12: Applying options for *
debug3: kex names ok: [curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1]
debug3: macs ok: [hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-ripemd160-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,hmac-ripemd160]
debug2: ssh_connect: needpriv 0
debug1: Connecting to 127.0.0.1 [127.0.0.1] port 2222.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/id_rsa type -1
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.8
debug1: Remote protocol version 2.0, remote software version mod_sftp/0.9.9
debug1: no match: mod_sftp/0.9.9
debug2: fd 3 setting O_NONBLOCK
debug3: put_host_port: [127.0.0.1]:2222
debug3: load_hostkeys: loading entries for host "[127.0.0.1]:2222" from file "/root/.ssh/known_hosts"
debug3: load_hostkeys: found key type RSA in file /root/.ssh/known_hosts:17
debug3: load_hostkeys: loaded 1 keys
debug3: load_hostkeys: loading entries for host "[127.0.0.1]:2222" from file "/etc/ssh/ssh_known_hosts"
debug3: load_hostkeys: loaded 0 keys
debug3: order_hostkeyalgs: prefer hostkeyalgs: ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-dss-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,ssh-dss
debug2: kex_parse_kexinit: chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
debug2: kex_parse_kexinit: chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
debug2: kex_parse_kexinit: hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-ripemd160-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,hmac-ripemd160
debug2: kex_parse_kexinit: hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-ripemd160-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,hmac-ripemd160
debug2: kex_parse_kexinit: zlib@openssh.com,zlib,none
debug2: kex_parse_kexinit: zlib@openssh.com,zlib,none
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1,rsa1024-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes256-ctr,aes192-ctr,aes128-ctr,aes256-cbc,aes192-cbc,aes128-cbc,blowfish-ctr,blowfish-cbc,cast128-cbc,arcfour256,arcfour128,3des-ctr,3des-cbc
debug2: kex_parse_kexinit: aes256-ctr,aes192-ctr,aes128-ctr,aes256-cbc,aes192-cbc,aes128-cbc,blowfish-ctr,blowfish-cbc,cast128-cbc,arcfour256,arcfour128,3des-ctr,3des-cbc
debug2: kex_parse_kexinit: hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-sha1-96,hmac-md5,hmac-md5-96,hmac-ripemd160,umac-64@openssh.com
debug2: kex_parse_kexinit: hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-sha1-96,hmac-md5,hmac-md5-96,hmac-ripemd160,umac-64@openssh.com
debug2: kex_parse_kexinit: zlib@openssh.com,zlib,none
debug2: kex_parse_kexinit: zlib@openssh.com,zlib,none
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: setup hmac-sha2-512
debug1: kex: server->client aes256-ctr hmac-sha2-512 zlib@openssh.com
debug2: mac_setup: setup hmac-sha2-512
debug1: kex: client->server aes256-ctr hmac-sha2-512 zlib@openssh.com
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<8192<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug2: bits set: 4072/8192
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: RSA ec:4e:57:e4:28:bc:88:93:8c:8b:71:1a:f2:ba:ca:5d
debug3: put_host_port: [127.0.0.1]:2222
debug3: put_host_port: [127.0.0.1]:2222
debug3: load_hostkeys: loading entries for host "[127.0.0.1]:2222" from file "/root/.ssh/known_hosts"
debug3: load_hostkeys: found key type RSA in file /root/.ssh/known_hosts:17
debug3: load_hostkeys: loaded 1 keys
debug3: load_hostkeys: loading entries for host "[127.0.0.1]:2222" from file "/etc/ssh/ssh_known_hosts"
debug3: load_hostkeys: loaded 0 keys
debug1: Host '[127.0.0.1]:2222' is known and matches the RSA host key.
debug1: Found key in /root/.ssh/known_hosts:17
debug2: bits set: 4120/8192
debug1: ssh_rsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
Bad packet length 1614781056.
Disconnecting: Packet corrupt
Write failed: Broken pipe
Couldn't read packet: Connection reset by peer

Strace output:

[pid  8896] 14:18:25.601185 read(3, "\0\0\0,\10\1\0\0\0\v\0\0\0\21Application error\0\0\0\5en-US\305\1\335h\16p\246<", 8192) = 48
[pid  8896] 14:18:25.601283 write(2, "Bad packet length 3941848338.\r\n", 31Bad packet length 3941848338.

On the server side:

2017-07-28 22:38:13,737 mod_sftp/0.9.9[24363]: received client version 'SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.8'
2017-07-28 22:38:13,737 mod_sftp/0.9.9[24363]: handling connection from SSH2 client 'OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.8'
2017-07-28 22:38:13,738 mod_sftp/0.9.9[24363]:  + Session key exchange: diffie-hellman-group-exchange-sha256
2017-07-28 22:38:13,738 mod_sftp/0.9.9[24363]:  + Session server hostkey: ssh-rsa
2017-07-28 22:38:13,738 mod_sftp/0.9.9[24363]:  + Session client-to-server encryption: aes256-ctr
2017-07-28 22:38:13,738 mod_sftp/0.9.9[24363]:  + Session server-to-client encryption: aes256-ctr
2017-07-28 22:38:13,738 mod_sftp/0.9.9[24363]:  + Session client-to-server MAC: hmac-sha2-512
2017-07-28 22:38:13,738 mod_sftp/0.9.9[24363]:  + Session server-to-client MAC: hmac-sha2-512
2017-07-28 22:38:13,738 mod_sftp/0.9.9[24363]:  + Session client-to-server compression: zlib@openssh.com
2017-07-28 22:38:13,738 mod_sftp/0.9.9[24363]:  + Session server-to-client compression: zlib@openssh.com
2017-07-28 22:38:13,904 mod_sftp/0.9.9[24363]: error setting key/IV for aes256-ctr cipher for decryption:
  (1) error:0906D06C:PEM routines:PEM_read_bio:no start line
  (2) error:0607B083:digital envelope routines:EVP_CipherInit_ex:no cipher set

What I Expected/Wanted

Be able to login :-)

ProFTPD Version and Configuration

Proftpd version (tried with 1.3.5d and 1.3.5e too)

Compile-time Settings:
  Version: 1.3.5e (maint)
  Platform: LINUX [Linux 3.13.0-112-generic x86_64]
  Built: Fri Jul 28 2017 22:29:07 UTC
  Built With:
    configure  '--prefix=/usr' '--mandir=/usr/share/man' '--sysconfdir=/etc/proftpd' '--localstatedir=/var/run' '--libexecdir=/usr/lib/proftpd' '--enable-devel=stacktrace' '--enable-sendfile' '--enable-facl' '--enable-dso' '--enable-autoshadow' '--enable-ctrls' '--enable-openssl' '--with-modules=mod_sql:mod_sql_passwd:mod_sql_mysql:mod_sftp:mod_sftp_sql' '--enable-nls' '--build' 'x86_64-linux-gnu' 'build_alias=x86_64-linux-gnu' 'CFLAGS=-O2'

  CFLAGS:  -Wall -DPR_DEVEL_STACK_TRACE -g -O0 -Wcast-align -Wchar-subscripts -Winline -Wstrict-prototypes -Wmissing-declarations -Wnested-externs -Wpointer-arith -Wshadow -Wundef -Wfloat-equal -Wformat -Wformat-security -Wimplicit-function-declaration -Wmaybe-uninitialized -Wpointer-to-int-cast -Wstack-protector -Wunreachable-code -fstack-protector-all
  LDFLAGS: -L$(top_srcdir)/lib   -L/usr/lib/x86_64-linux-gnu
  LIBS:  -lssl -lcrypto -lssl -lcrypto -lcap  -lm -lmysqlclient  -lpam  -lcrypto -lz -lsupp -lcrypt -ldl

  Files:
    Configuration File:
      /etc/proftpd/proftpd.conf
    Pid File:
      /var/run/proftpd.pid
    Scoreboard File:
      /var/run/proftpd.scoreboard
    Header Directory:
      /usr/include/proftpd
    Shared Module Directory:
      /usr/lib/proftpd

  Features:
    + Autoshadow support
    + Controls support
    + curses support
    + Developer support
    + DSO support
    + IPv6 support
    + Largefile support
    - Lastlog support
    - Memcache support
    + ncurses support
    + NLS support
    + OpenSSL support
    - PCRE support
    + POSIX ACL support
    + Shadow file support
    + Sendfile support
    + Trace support

  Tunable Options:
    PR_TUNABLE_BUFFER_SIZE = 1024
    PR_TUNABLE_DEFAULT_RCVBUFSZ = 8192
    PR_TUNABLE_DEFAULT_SNDBUFSZ = 8192
    PR_TUNABLE_GLOBBING_MAX_MATCHES = 100000
    PR_TUNABLE_GLOBBING_MAX_RECURSION = 8
    PR_TUNABLE_HASH_TABLE_SIZE = 40
    PR_TUNABLE_NEW_POOL_SIZE = 512
    PR_TUNABLE_SCOREBOARD_BUFFER_SIZE = 80
    PR_TUNABLE_SCOREBOARD_SCRUB_TIMER = 30
    PR_TUNABLE_SELECT_TIMEOUT = 30
    PR_TUNABLE_TIMEOUTIDENT = 10
    PR_TUNABLE_TIMEOUTIDLE = 600
    PR_TUNABLE_TIMEOUTLINGER = 30
    PR_TUNABLE_TIMEOUTLOGIN = 300
    PR_TUNABLE_TIMEOUTNOXFER = 300
    PR_TUNABLE_TIMEOUTSTALLED = 3600
    PR_TUNABLE_XFER_SCOREBOARD_UPDATES = 10

Conf file:

ServerName          test
ServerType          standalone
DefaultServer       on

# Disable standard ftp (we only use sftp, see below)
Port                0

# Don't use IPv6 support by default.
UseIPv6             off

# Umask 022 is a good standard umask to prevent new dirs and files
# from being group and world writable.
Umask               022

# To prevent DoS attacks, set the maximum number of child processes
# to 30.  If you need to allow more than 30 concurrent connections
# at once, simply increase this value.  Note that this ONLY works
# in standalone mode, in inetd mode you should use an inetd server
# that allows you to limit maximum number of processes per service
# (such as xinetd).
MaxInstances        30

# Set the user and group under which the server will run.
User                nobody
Group               nogroup

# create a user's home directory on demand if it doesn't exist
CreateHome          on 755 skel /etc/proftpd/skel dirmode 755 uid ~ gid ~

# Cause every FTP user to be "jailed" (chrooted) into their home
# New ftp users
# ToDo: use the commented lines when all users have been migrated to dedicated subfolder
DefaultRoot         ~ ftpgroup
DefaultChdir        ~ ftpgroup
#DefaultRoot         /home/proftpd/%u[0]/%u[0]%u[1]/%u ftpgroup
#DefaultChdir        /home/proftpd/%u[0]/%u[0]%u[1]/%u ftpgroup

# Cause every FTP user to be "jailed" (chrooted) into their home
# Existing ftp users
DefaultRoot         ~/sftp filetransfer
DefaultChdir        ~/sftp filetransfer

# Do not parse any files in the encountered directories called ".ftpaccess"
AllowOverride       off

# ToDo: remove AllowOverwrite when all sftp users are using dedicated folders
# Allow newly transfered files to overwrite existing files.
AllowOverwrite      on

# Bar use of SITE CHMOD by default
<Limit SITE_CHMOD>
    DenyAll
</Limit>

RootLogin         off
RequireValidShell off
TransferLog       /var/log/proftpd/xfer.log
SystemLog         /var/log/proftpd/proftpd.log
SyslogLevel       debug # emerg|alert|crit|error|warn|notice|info|debug

# FIFO setup
LogFormat         action_user_filepath_status "%m:%u:%f:%s:%S"
ExtendedLog       /var/log/proftpd/files.fifo WRITE action_user_filepath_status
ExtendedLog       /var/log/proftpd/files.log WRITE action_user_filepath_status

ExtendedLog       /var/log/proftpd/extended.log ALL

# Required as /var/log/wtmp is missing on Ubuntu cloud image
WtmpLog off

# Give access only to users in the "ftpgroup" we created
<Limit LOGIN>
    DenyGroup AND !ftpgroup,!filetransfer
</Limit>

<Directory "/">
    # Deny all permissions everywhere by default
    <Limit ALL>
        DenyAll
    </Limit>
</Directory>

# Obsolete folder (to be removed from proftpd conf)
# Every user uploads were put in that folder previously
# We are now using a dedicated subfolder for each user, see below
<Directory "/home/proftpd/sftp/upload">
    # All the files/folders except "/" are hidden
    HideFiles    !^/$
    HideNoAccess on

    <IfModule mod_rename.c>
        RenamePrefix ~.
    </IfModule>

    # Deny all permissions by default
    <Limit ALL>
        DenyAll
    </Limit>

    # REALPATH OPENDIR READDIR are required by Filezilla
    <Limit PWD REALPATH OPENDIR READDIR>
        IgnoreHidden on
        AllowAll
    </Limit>

    <Limit STOR>
        AllowAll
    </Limit>
</Directory>

# Upload folder for new ftp users (sftp with proftpd)
<Directory "/home/proftpd/%u[0]/%u[0]%u[1]/%u">
    HideFiles    "none"

    <IfModule mod_rename.c>
        RenameEnable off
    </IfModule>

    # Deny all permissions by default
    <Limit ALL>
        DenyAll
    </Limit>

    # Allow listing of files/directories
    # REALPATH OPENDIR READDIR are required by Filezilla
    <Limit PWD REALPATH OPENDIR READDIR MLSD LIST NLST CWD XCWD STAT LSTAT>
        AllowAll
    </Limit>
</Directory>

<Directory "/home/proftpd/%u[0]/%u[0]%u[1]/%u/inbound">
    # Do not allow newly transfered files to overwrite existing files.
    AllowOverwrite off

    # enable two-step file uploads: files are uploaded as ".in.filename."
    # and once the upload is complete, renamed to just filename.
    # (from v1.3.6 values can use the %P variable, which will be substituted with the session PID)
    # When HiddenStores is enabled, then DeleteAbortedStores is automatically enabled as well.
    # The FTP REST command is automatically blocked when HiddenStores is enabled.
    # HiddenStores .in. .%P # Use this when using version >= 1.3.6
    HiddenStores .in.

    # File size limit
    MaxStoreFileSize 20 Mb

    # Transfer rate limit (512kb/s but the throttling is only activated after the first 1024 bits)
    TransferRate STOR 512:1024

    <IfModule mod_rename.c>
        RenameEnable off
    </IfModule>

    # Deny all permissions by default
    <Limit ALL>
        DenyAll
    </Limit>

    # Allow listing of files/directories
    # REALPATH OPENDIR READDIR are required by Filezilla
    <Limit PWD REALPATH OPENDIR READDIR MLSD LIST NLST CWD XCWD STAT LSTAT>
        AllowAll
    </Limit>

    # Allow upload
    <Limit STOR>
        AllowAll
    </Limit>
</Directory>

<Directory "/home/proftpd/%u[0]/%u[0]%u[1]/%u/outbound">
    # Permit clients from performing "restart" download (retrieve) file transfers
    AllowRetrieveRestart on

    <IfModule mod_rename.c>
        RenameEnable off
    </IfModule>

    # Deny all permissions by default
    <Limit ALL>
        DenyAll
    </Limit>

    # Allow listing of files/directories
    # REALPATH OPENDIR READDIR are required by Filezilla
    <Limit PWD REALPATH OPENDIR READDIR MLSD LIST NLST CWD XCWD STAT LSTAT>
        AllowAll
    </Limit>

    # Allow download
    <Limit RETR>
        AllowAll
    </Limit>
</Directory>

# Folders for existing ftp users (previously sftp with ssh server, no restriction)
<Directory "~/sftp">
    # No limit
    <Limit ALL>
        AllowAll
    </Limit>
</Directory>

# ToDo: remove the rename module when all sftp users are using dedicated folders
# Enable rename module.
<IfModule mod_rename.c>
    RenameEngine on
    RenameLog /var/log/proftpd/rename.log
</IfModule>

<IfModule mod_sql.c>
    # Hash settings
    SQLPasswordEngine on
    SQLNamedQuery get-user-salt SELECT "`salt` FROM `ftpuser` WHERE `userid` = '%{0}' AND (`authentication` = 'PASSWORD' OR `authentication` = 'BOTH') AND isactive=1"
    SQLPasswordUserSalt sql:/get-user-salt Prepend

    # MySQL settings
    SQLBackend      mysql

    SQLAuthenticate on

    # Use both a crypted or plaintext password
    SQLAuthTypes SHA256 Crypt

    # DB Connection
    SQLConnectInfo ftp@xxxx xxx xxxxx

    # Describes both users/groups tables
    SQLUserInfo ftpuser userid passwd uid gid homedir shell
    SQLGroupInfo ftpgroup groupname gid members

    # set min UID and GID - otherwise these are 999 each
    SQLMinID        500

    # Update count every time user logs in
    SQLLog PASS updatecount
    SQLNamedQuery updatecount UPDATE "count=count+1, accessed=now() WHERE userid='%u'" ftpuser

    # Update modified everytime user uploads or deletes a file
    SQLLog  STOR,DELE modified
    SQLNamedQuery modified UPDATE "modified=now() WHERE userid='%u'" ftpuser

    SQLNamedQuery get-user-authorized-keys SELECT "`sshKey` FROM `userkeys` INNER JOIN `ftpuser` ON `userkeys`.`userId` = `ftpuser`.`userid` AND (`ftpuser`.`authentication` = 'SSH' OR `ftpuser`.`authentication` = 'BOTH') WHERE `userkeys`.`userId`='%U' AND `ftpuser`.`isactive`=1"

    SqlLogFile /var/log/proftpd/sql.log
</IfModule>

<IfModule mod_sftp.c>
    # Enables the SFTP ability for the server.
    SFTPEngine on

    # Specifies the port where the SFTP connections will be accepted.
    Port 2222

    # Configures the location of the log file that will be created.
    SFTPLog /var/log/proftpd/sftp.log

    # Configure both the RSA and DSA host keys, using the same host key files that OpenSSH uses.
    SFTPHostKey /etc/ssh/ssh_host_rsa_key
    SFTPHostKey /etc/ssh/ssh_host_dsa_key

    # This line configures the server to only accept connections with SSH keys.
    SFTPAuthMethods publickey password

    # Enable compression
    SFTPCompression delayed

    # Disable all unsafe sftp options
    # See http://www.proftpd.org/docs/contrib/mod_sftp.html#SFTPOptions
    SFTPOptions IgnoreSFTPUploadPerms
    SFTPOptions IgnoreSCPUploadTimes
    SFTPOptions IgnoreSFTPSetOwners
    SFTPOptions IgnoreSFTPSetPerms
    SFTPOptions IgnoreSFTPSetTimes
    SFTPOptions IgnoreSFTPUploadPerms

    # Workaround for Coles (SSH-2.0-SharpSSH-1.1.1.13-JSCH-0.1.28)
    SFTPClientMatch ".*SharpSSH.*" channelWindowSize 1GB

    <IfModule mod_sftp_sql.c>
        SFTPAuthorizedUserKeys sql:/get-user-authorized-keys
    </IfModule>
</IfModule>

The same setup is definitely working with OpenSSL 1.0.1f

[Vincent--]

Just tested with 1.3.6 and it seems to be ok. Not sure 1.3.6 is production ready however...
Thoughts about that? Is it possible to fix the issue in 1.3.5 branch?

[Vincent--]

I cannot use v1.3.6 for now due to Castaglia/proftpd-mod_rename#4

[Vincent--]

Thanks for the fix for mod_rename and for your responsiveness, I really appreciate that 👍
About this issue, I use openssl 1.1.x from https://launchpad.net/~ondrej/+archive/ubuntu/php (if that helps)

working version:

OpenSSL 1.0.1f 6 Jan 2014
built on: Mon Jan 30 20:38:38 UTC 2017
platform: debian-amd64
options:  bn(64,64) rc4(16x,int) des(idx,cisc,16,int) blowfish(idx)
compiler: cc -fPIC -DOPENSSL_PIC -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -m64 -DL_ENDIAN -DTERMIO -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wa,--noexecstack -Wall -DMD32_REG_T=int -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM
OPENSSLDIR: "/usr/lib/ssl"

Non working version:

OpenSSL 1.1.0f  25 May 2017
built on: reproducible build, date unspecified
platform: debian-amd64
compiler: gcc -DDSO_DLFCN -DHAVE_DLFCN_H -DNDEBUG -DOPENSSL_THREADS -DOPENSSL_NO_STATIC_ENGINE -DOPENSSL_PIC -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DRC4_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DPADLOCK_ASM -DPOLY1305_ASM -DOPENSSLDIR="\"/usr/lib/ssl\"" -DENGINESDIR="\"/usr/lib/x86_64-linux-gnu/engines-1.1\""
OPENSSLDIR: "/usr/lib/ssl"
ENGINESDIR: "/usr/lib/x86_64-linux-gnu/engines-1.1"

There might be other differences between the 2 systems but this is the only one I've spotted

[Castaglia]

I am thinking that you are encountering Bug#4240; the OpenSSL 1.1.x API changed -- and ProFTPD 1.3.5 does not have the corresponding code changes for that newer OpenSSL version.

You can see for yourself what changes may be needed:

$ git clone https://github.com/proftpd/proftpd.git
$ cd proftpd/
$ git diff v1.3.5e v1.3.6 -- contrib/mod_sftp/cipher.c

With the release of ProFTPD 1.3.6 stable, though, there will not be any backported changes to the 1.3.5 branch; see the Versioning doc.

[Vincent--]

Fair enough about the versioning model but in that case, that really seems to be a critical bug as the application is just completely unusable. At least a warning about that should be added in the doc maybe...

I'm testing v.1.3.6 and will upgrade if everything is ok.
I'll let you know if I found any issue with this version.

[Vincent--]

Happy to close this issue as 1.3.6 is working with last version of openssl and the FAQ has been updated for previous version < 1.3.6

[Castaglia]

Great, will do. Thanks!