Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] [FTP] The connections of SFTP didn't release when task failed. #1210

Closed
2 of 3 tasks
FlechazoW opened this issue Sep 1, 2022 · 1 comment · Fixed by #1222
Closed
2 of 3 tasks

[Bug] [FTP] The connections of SFTP didn't release when task failed. #1210

FlechazoW opened this issue Sep 1, 2022 · 1 comment · Fixed by #1222
Labels
bug Something isn't working good first issue Good for newcomers
Projects

Comments

@FlechazoW
Copy link
Member

Search before asking

  • I had searched in the issues and found no similar issues.

What happened

When the task fails due to issues like 'no such file' or 'file is empty', the sftp connection didn't release.

Task mode is yarn-session.

image

What you expected to happen

Connection is released.

How to reproduce

  1. Create a task which source is 'sftp'.
  2. The file that need to be synced is empty or not exist.
{
  "job": {
    "content": [
      {
        "reader": {
          "parameter": {
            "path": "/root/tiezhu/test_two.csv",
            "protocol": "SFTP",
            "password": "Abc!@#135",
            "port": 22,
            "isFirstLineHeader": true,
            "host": "172.16.85.171",
            "column": [
              {
                "name": "id",
                "type": "int"
              },
              {
                "name": "name",
                "type": "string"
              }
            ],
            "fieldDelimiter": ",",
            "encoding": "utf-8",
            "fileType": "CSV",
            "username": "root"
          },
          "name": "ftpreader"
        },
        "writer": {
          "parameter": {
            "print": true
          },
          "name": "streamwriter"
        }
      }
    ]
  }
}

Anything else

NONE

Version

master

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@FlechazoW FlechazoW added the bug Something isn't working label Sep 1, 2022
@FlechazoW FlechazoW added this to Needs triage in roadmap via automation Sep 1, 2022
@FlechazoW FlechazoW added the good first issue Good for newcomers label Sep 1, 2022
@FlechazoW
Copy link
Member Author

same bug with local test mode.

{
  "job": {
    "content": [
      {
        "reader": {
          "parameter": {
            "password": "DT@Stack#123",
            "column": [
              {
                "name": "id",
                "type": "INT"
              },
              {
                "name": "result_val",
                "type": "VARCHAR"
              }
            ],
            "connection": [
              {
                "schema": "flinkx_test",
                "jdbcUrl": [
                  "jdbc:mysql://172.16.100.186:3306/flinkx_test?useSSL=false"
                ],
                "table": [
                  "ftp_source"
                ]
              }
            ],
            "splitPk": "id",
            "username": "drpeco"
          },
          "name": "mysqlreader"
        },
        "writer": {
          "parameter": {
            "path": "/data/sftp/buffertest",
            "password": "dt@sz.com",
            "protocol": "SFTP",
            "port": "22",
            "host": "172.16.100.116",
            "column": [
              {
                "name": "id",
                "type": "INT"
              },
              {
                "name": "result_val",
                "type": "VARCHAR"
              }
            ],
            "writeMode": "append",
            "fieldDelimiter": ",",
            "ftpFileName": "xiaohe.txt",
            "username": "root"
          },
          "name": "ftpwriter",
          "table": {
            "tableName": "sink"
          }
        }
      }
    ],
    "setting": {
      "speed": {
        "bytes": 0,
        "channel": 2
      }
    }
  }
}

limengyao0809 added a commit to limengyao0809/chunjun that referenced this issue Sep 6, 2022
roadmap automation moved this from Needs triage to Closed Sep 6, 2022
FlechazoW pushed a commit that referenced this issue Sep 6, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working good first issue Good for newcomers
Projects
roadmap
Closed
1 participant