Implement and verify File Integration use cases

[kasun04]

• File protocols (FTP, SFTP, local)
• File consumers (consuming based on a delimiter, streaming files)
• File producer/writers

[Nipunich]

Following table shows the list of file integration use cases that needs to be verified and implemented.

File Integration Use Cases

Use Case	Sub Use case	Ballerina FTP Module / File Module	Status
Reading a file from a specified location in file system
	From Local location	✅	improve
	From FTP location	✅
	From FTP location in passive mode	✅
	From SFTP location	✅
	From SFTP location in passive mode	✅
	From FTPS location in passive mode
Writing a file to a specified location in file system
	To Local location	✅	improve
	To FTP location	✅
	To FTP location in passive mode	✅
	To SFTP location	✅
	To SFTP location in passive mode	✅
	To FTPS location in passive mode
Moving a file without a transformation		✅
	Move a file/folder within the same server
Get file properties
	Get size of a file	✅
	Get last modified time of a file
Split and merge files		✅
	Splits files into multiple chunks
	Merges files into a single file
	Append content to an existing file
Read specified lines of a file
	Read specific lines between given line numbers from a file
	Reads a specific line from a file
Processing archived files
	Archive files or folder in the file system
	List all files inside zip file
	Decompresses a zip file in the file system
Check and Fetch files in the file system
	Search for a file based on a given file pattern, directory pattern in a specified location
	Checks the existence of a file in a specified location.
	Fetch a file by a pattern
	Get the content list of a given path	✅
Copy files from one location to another
	Copy particular files with given file pattern from files on the local physical file system or a file on an FTP server to a specified location
Create a file or folder
	Create a file or folder on the local physical file system with or without content
	Create a file or folder on a FTP server with or without content	✅
Deleting a file or folder
	Delete a file / folder on the local physical file system
	Delete a file / folder on the FTP server	✅
Sending a file
	Send a file to a specified location on the local physical file system
	Send a file to a specified location on a ftp server	✅
Connect to a FTP server through a Proxy
	Connect to a ftp server through a proxy

[Nipunich]

I have drafted a sample program to implement the functionality of the file listener listening to a local folder. Here when a modification is done to the listening folder, the files are processed and move them to a specified location.

import ballerina/file;
import ballerina/log;
import ballerina/internal;
import ballerina/io;

listener file:Listener localFolder = new ({
    path: "/Users/nipuni/Documents/Ballerina_Integrator/file-integration/fs",
    recursive: false
});

service fileSystem on localFolder {

    resource function onModify(file:FileEvent m) {

        error? e = localFolder.__start();
        log:printInfo("File created");

        string source = "/Users/nipuni/Documents/Ballerina_Integrator/file-integration/fs";
        string fileNamePattern = "xml";

        internal:Path src = new(source);

        internal:Path[]|error fileList = src.list();

        if (fileList is internal:Path[]) {
            foreach (var file in fileList) {
                string ext = file.getExtension();

                if (ext == fileNamePattern) {
                    string name = file.getName();

                    string srcPath = "/Users/nipuni/Documents/Ballerina_Integrator/file-integration/fs";
                    srcPath = srcPath + "/" + name;

                    string target = "/Users/nipuni/Documents/Ballerina_Integrator/file-integration/out";
                    target = target + "/" + name;

                    internal:Path srcP = new(srcPath);
                    internal:Path dest = new(target);

                    processFile(ext, srcPath);
                    log:printInfo("Processing finished");

                    error? er = srcP.moveTo(dest);
                    io:println(er);
                    log:printInfo("Operation successful");

                }
            }
        }
    }
}

// User can define the the processing ( Following is an example).
public function processFile(string ext, string source) {
    log:printInfo("Started processing the file ");

    if (ext == "xml") {
        log:printInfo("Reading xml file");
        io:ReadableByteChannel rbc = io:openReadableFile(source);
        io:println("Get byte channel");
        io:ReadableCharacterChannel rch = new(rbc, "UTF8");
        var result = rch.readXml();

        if (result is xml) {
            json j1 = result.toJSON({
            
            });
        }
    }

    if (ext == "csv") {
        log:printInfo("Reading csv file");
        io:ReadableCSVChannel rCsvChannel = io:openReadableCsvFile(source);

        while (rCsvChannel.hasNext()) {
            var records = rCsvChannel.getNext();

            if (records is string[]) {
                io:println(records);
            }
        }
    }
}

[Nipunich]

Following util functions are implemented to process xml and json files.

// Copyright (c) 2019 WSO2 Inc. (http://www.wso2.org) All Rights Reserved.
//
// WSO2 Inc. licenses this file to you under the Apache License,
// Version 2.0 (the "License"); you may not use this file except
// in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied.  See the License for the
// specific language governing permissions and limitations
// under the License.

import ballerina/log;
import ballerina/io;
import ballerina/config;
import ballerina/http;

// Util function to read a json file.
public function readJsonFile(io:ReadableByteChannel result) returns json|error {

    io:ReadableCharacterChannel? charChannelResult = getCharChannel(result);
    var resultJson = charChannelResult.readJson();

    if (resultJson is json) {
        io:println("File content: ", resultJson);
        return resultJson;
    } else {
        log:printError("An error occured.", err = resultJson);
        return resultJson;
    }
}

// Util function to read a xml file.
public function readXmlFile(io:ReadableByteChannel result) returns xml|error? {

    io:ReadableCharacterChannel? charChannelResult = getCharChannel(result);
    var resultXml = charChannelResult.readXml();

    if (resultXml is xml) {
        io:println("File content: ", resultXml);
        return resultXml;
    } else {
        log:printError("An error occured.", err = resultXml);
        return resultXml;
    }
}

// Util function to convert a byte channel to a character channel.
public function getCharChannel(io:ReadableByteChannel getResult) returns io:ReadableCharacterChannel? {

    io:ReadableCharacterChannel? charChannel = new io:ReadableCharacterChannel(getResult, "utf-8");

    if (charChannel is io:ReadableCharacterChannel) {
        return charChannel;
    } else {
        log:printError("An error occured.");
        return;
    }
}