wget command can be downloaded, but curl command is not downloaded #29

sanshengshui · 2023-10-24T09:36:33Z

As follows:

pom.xml:

<!--ftp-->
<dependency>
  <groupId>org.apache.ftpserver</groupId>
  <artifactId>ftpserver-core</artifactId>
  <version>1.1.1</version>
</dependency>

elecharny · 2023-10-24T09:59:56Z

Hi,

have you tried with ftpserver 1.1.4?

sanshengshui · 2023-10-24T11:53:23Z

Ok, I'll try👌

By the way, what's the difference between these two versions？

elecharny · 2023-10-24T20:46:32Z

1.1.2: https://issues.apache.org/jira/projects/FTPSERVER/versions/12340451
1.1.3 and 1.1.4 were mainly bumping up some dependencies.

I'd just like to know if you still have the issue with a maintained version, in order to investigate it with the latest code base.

thanks!

sanshengshui · 2023-10-26T08:46:24Z

I just upgraded to 1.1.4
Here's what happens when you download with curl

➜  Downloads curl -u 0297XfYHmj:Cbr3QqJC -o ocpp.tar.gz ftp://f1.iot17.com:8183/1698304726c2bcfc8265c.tar.gz -v
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 198.19.10.106:8183...
* Connected to f1.iot17.com (198.19.10.106) port 8183 (#0)
< 220 Service ready for new user.
> USER 0297XfYHmj
< 331 User name okay, need password for 0297XfYHmj.
> PASS Cbr3QqJC
< 230 User logged in, proceed.
> PWD
< 257 "/" is current directory.
* Entry path is '/'
* Request has same path as previous transfer
> EPSV
* Connect data stream passively
* ftp_perform ends with SECONDARY: 0
< 229 Entering Passive Mode (|||13005|)
*   Trying 198.19.10.106:13005...
* Connecting to 198.19.10.106 (198.19.10.106) port 13005
* Connected to f1.iot17.com (198.19.10.106) port 8183 (#0)
> TYPE I
< 200 Command TYPE okay.
> SIZE 1698304726c2bcfc8265c.tar.gz
< 213 16483570
> RETR 1698304726c2bcfc8265c.tar.gz
  0 15.7M    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0< 150 File status okay; about to open data connection.
* Maxdownload = -1
* Getting file with size: 16483570
  0 15.7M    0     0    0     0      0      0 --:--:--  0:00:04 --:--:--     0{ [0 bytes data]
  0 15.7M    0     0    0     0      0      0 --:--:--  0:00:05 --:--:--     0* transfer closed with 16483570 bytes remaining to read
  0 15.7M    0     0    0     0      0      0 --:--:--  0:00:05 --:--:--     0
* Closing connection 0
curl: (18) transfer closed with 16483570 bytes remaining to read

It's a different phenomenon than before，What are the possible reasons?

elecharny · 2023-10-26T09:19:21Z

Can you check if you have any log on the FtpServer side?

More specifically, any log containing the string "Exception during data transfer, closing data connection socket".

sanshengshui · 2023-10-26T10:25:36Z

The error message is as follows:

2023-10-26 16:41:01.018|TID:N/A|WARN |org.apache.ftpserver.impl.IODataConnectionFactory|createDataSocket|351|pool-18-thread-594| FtpDataConnection.getDataSocket()
java.net.SocketTimeoutException: Accept timed out
        at java.base/java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.base/java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:458)
        at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:565)
        at java.base/java.net.ServerSocket.accept(ServerSocket.java:533)
        at org.apache.ftpserver.impl.IODataConnectionFactory.createDataSocket(IODataConnectionFactory.java:329)
        at org.apache.ftpserver.impl.IODataConnectionFactory.openConnection(IODataConnectionFactory.java:231)
        at com.tuya.sftp.protocol.gateway.service.protocol.command.RetrCommand.execute(RetrCommand.java:74)
        at org.apache.ftpserver.impl.DefaultFtpHandler.messageReceived(DefaultFtpHandler.java:211)
        at org.apache.ftpserver.listener.nio.FtpHandlerAdapter.messageReceived(FtpHandlerAdapter.java:62)
        at org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceived(DefaultIoFilterChain.java:1015)
        at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650)
        at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49)
        at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1128)
        at org.apache.ftpserver.listener.nio.FtpLoggingFilter.messageReceived(FtpLoggingFilter.java:85)
        at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650)
        at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49)
        at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1128)
        at org.apache.mina.core.filterchain.IoFilterEvent.fire(IoFilterEvent.java:106)
        at org.apache.mina.filter.logging.MdcInjectionFilter.filter(MdcInjectionFilter.java:162)
        at org.apache.mina.filter.util.CommonEventFilter.messageReceived(CommonEventFilter.java:84)
        at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650)
        at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49)
        at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1128)
        at org.apache.mina.filter.codec.ProtocolCodecFilter$ProtocolDecoderOutputImpl.flush(ProtocolCodecFilter.java:418)
        at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:257)
        at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650)
        at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49)
        at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1128)
        at org.apache.mina.core.filterchain.IoFilterEvent.fire(IoFilterEvent.java:106)
        at org.apache.mina.core.session.IoEvent.run(IoEvent.java:89)
        at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTask(OrderedThreadPoolExecutor.java:763)
        at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTasks(OrderedThreadPoolExecutor.java:755)
        at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.run(OrderedThreadPoolExecutor.java:695)
        at java.base/java.lang.Thread.run(Thread.java:829)

elecharny · 2023-10-26T10:54:14Z

Ok, this is clearly not a problem in MINA FTP server code. At some point, when the RETR command is received, the following method is called:

        at com.tuya.sftp.protocol.gateway.service.protocol.command.RetrCommand.execute(RetrCommand.java:74)

then it tries to open a connection to a remote peer which fails (the top of the stack trace).

I suggest you ask the third party (tuya.com) about what's going on there.

elecharny · 2023-10-26T11:24:48Z

Some question:

the fact the socket times out drives me to think it's more like a configuration issue on your side than a problem in the code. Can you check that?
Which product are you using? Is it a tuya.com API based lib?

sanshengshui · 2023-10-27T02:12:16Z

I am an employee of tuya.com and am trying to set up an ftp service using FTP-server

I'll keep trying to figure out what caused it

Whether wget enters epsv mode and curl defaults to pasv mode

elecharny · 2023-10-27T03:06:06Z

Ok, so can you provide the code associated with om.tuya.sftp.protocol.gateway.service.protocol.command.RetrCommand?

sanshengshui · 2023-10-27T03:15:26Z

I made a preliminary determination that curl went to epsv mode by default.
The curl command is as follows:

➜  ~ curl --ftp-pasv -u 02982C3LaV:TjTv7Swb -O ftp://f1.iot17.com:8183/1698304726c2bcfc8265c.tar.gz -v
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 198.19.10.106:8183...
* Connected to f1.iot17.com (198.19.10.106) port 8183 (#0)
  0     0    0     0    0     0      0      0 --:--:--  0:00:04 --:--:--     0< 220 Service ready for new user.
> USER 02982C3LaV
  0     0    0     0    0     0      0      0 --:--:--  0:00:05 --:--:--     0< 331 User name okay, need password for 02982C3LaV.
> PASS TjTv7Swb
< 230 User logged in, proceed.
> PWD
< 257 "/" is current directory.
* Entry path is '/'
* Request has same path as previous transfer
> EPSV
* Connect data stream passively
* ftp_perform ends with SECONDARY: 0
< 229 Entering Passive Mode (|||13017|)
*   Trying 198.19.10.106:13017...
* Connecting to 198.19.10.106 (198.19.10.106) port 13017
* Connected to f1.iot17.com (198.19.10.106) port 8183 (#0)
> TYPE I
< 200 Command TYPE okay.
> SIZE 1698304726c2bcfc8265c.tar.gz
< 213 16483570
> RETR 1698304726c2bcfc8265c.tar.gz
  0 15.7M    0     0    0     0      0      0 --:--:--  0:00:05 --:--:--     0< 150 File status okay; about to open data connection.
* Maxdownload = -1
* Getting file with size: 16483570
  0 15.7M    0     0    0     0      0      0 --:--:--  0:00:09 --:--:--     0{ [0 bytes data]
* transfer closed with 16483570 bytes remaining to read
  0 15.7M    0     0    0     0      0      0 --:--:--  0:00:10 --:--:--     0
* Closing connection 0
curl: (18) transfer closed with 16483570 bytes remaining to read

When I use wget, pasv mode works fine

➜  ~ wget ftp://02982C3LaV:TjTv7Swb@f1.iot17.com:8183/1698304726c2bcfc8265c.tar.gz                    
--2023-10-27 11:10:17--  ftp://02982C3LaV:*password*@f1.iot17.com:8183/1698304726c2bcfc8265c.tar.gz
           => “1698304726c2bcfc8265c.tar.gz”
正在解析主机 f1.iot17.com (f1.iot17.com)... 198.19.10.106
正在连接 f1.iot17.com (f1.iot17.com)|198.19.10.106|:8183... 已连接。
正在以 02982C3LaV 登录 ... 登录成功！
==> SYST ... 完成。   ==> PWD ... 完成。
==> TYPE I ... 完成。 ==> 不需要 CWD。
==> SIZE 1698304726c2bcfc8265c.tar.gz ... 16483570
==> PASV ... 完成。   ==> RETR 1698304726c2bcfc8265c.tar.gz ... 完成。
长度：16483570 (16M) (非正式数据)

1698304726c2bcfc8265c.tar.gz          100%[======================================================================>]  15.72M  2.24MB/s  用时 7.3s    

2023-10-27 11:10:25 (2.16 MB/s) - “1698304726c2bcfc8265c.tar.gz” 已保存 [16483570]

my ftp-server code as follows:

        FtpServerFactory serverFactory = new FtpServerFactory();
        CommandFactoryFactory commandFactoryFactory = new CommandFactoryFactory();
        commandFactoryFactory.addCommand("STOR", new StorCommand());
        commandFactoryFactory.addCommand("RETR", new RetrCommand());
        commandFactoryFactory.addCommand("USER", new UserCommand());
        commandFactoryFactory.addCommand("PASS", new PassCommand());
        commandFactoryFactory.addCommand("CWD", new CwdCommand());
        commandFactoryFactory.addCommand("TYPE", new TypeCommand());
        commandFactoryFactory.addCommand("SIZE", new SizeCommand());
        commandFactoryFactory.addCommand("PASV", new PasvCommand());
        CommandFactory commandFactory = commandFactoryFactory.createCommandFactory();
        serverFactory.setCommandFactory(commandFactory);
        ListenerFactory listenerFactory = new ListenerFactory();
        listenerFactory.setPort(ftpConfig.getFtpPort());
        DataConnectionConfigurationFactory dataConnectionConfFactory = new DataConnectionConfigurationFactory();
        dataConnectionConfFactory.setPassivePorts(ftpConfig.getPassivePorts());
        String externalIP = ExternalIPUtils.getExternalIP(ftpConfig.getPublicIpv4Domain());
        ......
       Listener listener = listenerFactory.createListener();
        serverFactory.addListener("default", listener);
        Map<String, Ftplet> ftpLets = new HashMap();
        ftpLets.put("ftpService", new MyFtpPlet());
        serverFactory.setFtplets(ftpLets);
        ......
         server = serverFactory.createServer();

my RetrCommand as follows:

@Slf4j
public class RetrCommand extends RETR {

    @Override
    public void execute(FtpIoSession session, FtpServerContext context, FtpRequest request) {
        try {
            // argument check
            String fileName = request.getArgument();
            if (fileName == null) {
                session.write(LocalizedDataTransferFtpReply.translate(session, request,
                        context, FtpReply.REPLY_501_SYNTAX_ERROR_IN_PARAMETERS_OR_ARGUMENTS, "RETR", null, null));
                return;
            }

            // 24-10-2007 - added check if PORT or PASV is issued, see
            // https://issues.apache.org/jira/browse/FTPSERVER-110
            //TODO move this block of code into the super class. Also, it makes
            //sense to have this as the first check before checking everything
            //else such as the file and its permissions.
            DataConnectionFactory connFactory = session.getDataConnection();
            if (connFactory instanceof IODataConnectionFactory) {
                InetAddress address = ((IODataConnectionFactory) connFactory).getInetAddress();
                log.info("======>address:{}", JsonUtils.toString(address));
                if (address == null) {
                    session.write(new DefaultFtpReply(FtpReply.REPLY_503_BAD_SEQUENCE_OF_COMMANDS, "PORT or PASV must be issued first"));
                    return;
                }
            }

            // get data connection
            session.write(LocalizedFtpReply.translate(session, request, context, FtpReply.REPLY_150_FILE_STATUS_OKAY, "RETR", null));

            // send file data to client
            boolean failure = false;
            InputStream is = null;

            FtpFile file = null;
            DataConnection dataConnection;
            try {
                dataConnection = session.getDataConnection().openConnection();
            } catch (Exception e) {
                log.debug("Exception getting the output data stream", e);
                session.write(LocalizedDataTransferFtpReply.translate(session, request, context,
                        FtpReply.REPLY_425_CANT_OPEN_DATA_CONNECTION, "RETR", null, null));
                return;
            }
            long transSz = 0L;
            try {
                try {
                    User user = session.getUser();
                    fileName = fileName.replace("/", "");
                    String objectKey = user.getHomeDirectory();
                    String fileUrl = objectKey + fileName;
                    log.info("name:{},fileUrl:{}", user.getName(), fileUrl);
                    IFtpFileService ftpFileService = SpringHolder.getBean(IFtpFileService.class);
                    is = ftpFileService.openUrlInputStreamByCetus(user.getName(), fileUrl);
                } catch (Exception e) {
                    session.write(LocalizedDataTransferFtpReply.translate(session, request, context,
                            FtpReply.REPLY_425_CANT_OPEN_DATA_CONNECTION, "RETR", null, null));
                    e.printStackTrace();
                    return;
                }

                // transfer data
                transSz = dataConnection.transferToClient(session.getFtpletSession(), is);
                // attempt to close the input stream so that errors in
                // closing it will return an error to the client (FTPSERVER-119)
                if (is != null) {
                    is.close();
                }

                log.info("File downloaded {}", fileName);

                // notify the statistics component
                ServerFtpStatistics ftpStat = (ServerFtpStatistics) context
                        .getFtpStatistics();
                if (ftpStat != null) {
                    ftpStat.setDownload(session, file, transSz);
                }

            } catch (SocketException ex) {
                log.debug("Socket exception during data transfer", ex);
                failure = true;
                session.write(LocalizedDataTransferFtpReply.translate(session, request, context,
                        FtpReply.REPLY_426_CONNECTION_CLOSED_TRANSFER_ABORTED,
                        "RETR", fileName, null, transSz));
            } catch (IOException ex) {
                log.debug("IOException during data transfer", ex);
                failure = true;
                session.write(LocalizedDataTransferFtpReply.translate(
                        session, request, context, FtpReply.REPLY_551_REQUESTED_ACTION_ABORTED_PAGE_TYPE_UNKNOWN,
                        "RETR", fileName, file, transSz));
            } finally {
                // make sure we really close the input stream
                IoUtils.close(is);
            }

            // if data transfer ok - send transfer complete message
            if (!failure) {
                session.write(LocalizedDataTransferFtpReply.translate(session, request, context,
                        FtpReply.REPLY_226_CLOSING_DATA_CONNECTION, "RETR", fileName, file, transSz));

            }
        } finally {
            session.resetState();
            session.getDataConnection().closeDataConnection();
        }
    }
}

The key code is：
dataConnection = session.getDataConnection().openConnection();

sanshengshui · 2023-10-27T03:23:37Z

One way to cheat is: Through what configuration can the client only go to PASV mode
😄

elecharny · 2023-10-27T07:46:07Z

@sanshengshui the diff I see is that curl uses ESPV while wget uses PASV. Is that a IPV6 connection?

sanshengshui · 2023-10-27T08:52:28Z

IPV4 connection

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

wget command can be downloaded, but curl command is not downloaded #29

wget command can be downloaded, but curl command is not downloaded #29

sanshengshui commented Oct 24, 2023

elecharny commented Oct 24, 2023

sanshengshui commented Oct 24, 2023 •

edited

elecharny commented Oct 24, 2023

sanshengshui commented Oct 26, 2023

elecharny commented Oct 26, 2023

sanshengshui commented Oct 26, 2023

elecharny commented Oct 26, 2023

elecharny commented Oct 26, 2023

sanshengshui commented Oct 27, 2023

elecharny commented Oct 27, 2023

sanshengshui commented Oct 27, 2023 •

edited

sanshengshui commented Oct 27, 2023 •

edited

elecharny commented Oct 27, 2023

sanshengshui commented Oct 27, 2023

wget command can be downloaded, but curl command is not downloaded #29

wget command can be downloaded, but curl command is not downloaded #29

Comments

sanshengshui commented Oct 24, 2023

elecharny commented Oct 24, 2023

sanshengshui commented Oct 24, 2023 • edited

elecharny commented Oct 24, 2023

sanshengshui commented Oct 26, 2023

elecharny commented Oct 26, 2023

sanshengshui commented Oct 26, 2023

elecharny commented Oct 26, 2023

elecharny commented Oct 26, 2023

sanshengshui commented Oct 27, 2023

elecharny commented Oct 27, 2023

sanshengshui commented Oct 27, 2023 • edited

sanshengshui commented Oct 27, 2023 • edited

elecharny commented Oct 27, 2023

sanshengshui commented Oct 27, 2023

sanshengshui commented Oct 24, 2023 •

edited

sanshengshui commented Oct 27, 2023 •

edited

sanshengshui commented Oct 27, 2023 •

edited