Folders

The Folders node.js package provides a filesystem abstraction for synthetic file systems.

Folders may be based on a local file system, a remote file system, or another synthetic file system as provided by a module. Additional providers are available and can be installed via "npm install folders-modulename".

For example: "npm install folders-ftp" will install an FTP module. The module comes with an FTP client and server, enabling Folders to use Folders on a remote system, and to provide access to folders over the FTP protocol to clients.

You need to install the gulp tasker module by command "npm install --global gulp". Then you can go to src and run "gulp test". That will run the test directory test cases.

The core project is available under the Apache 2.0 or MIT licenses. Modules and their dependencies may have different license requirements.

Folders Provider API

Folders providers must implement the following methods:

Constructor

Provider constructor, could pass the special option/param in the opts param.

/**
 * @param Prefix, folders prefix
 * @param opts, options, example connectionString for ftp
 */
Provider(prefix, opts)

some options for exist folders provider.

• folders-ftp options

{
	// the connection string, format: ftp//username:password@host:port
	connectionString : "ftp://test:123456@localhost:3333",

	// the option to start up a embedded server when inin the folders, used in test/debug
	enableEmbeddedServer : true
}

• folders-ssh options

{
	// the connection string, format: ssh//username:password@host:port
	connectionString : "ssh://test:123456@localhost:3334",

	// the option to start up a embedded server when inin the folders, used in test/debug
	enableEmbeddedServer : true
}

• folders-hdfs options

{
	// the base url address for hdfs instance
	baseurl : "http://webhdfs.node/webhdfs/v1/data/",

	// the username to access the hdfs instances
	username : 'hdfs'
}

###ls

ls dir

/**
 * @param uri, the uri to ls
 * @param cb, callback function. 
 */
ls(uri, cb)

//the param of the cb function
/**
 * @param err, the err message, the files will be null if err, please check the err before using the files information.
 * @param files ,the file information if success
 */
cb(err, files);

###cat

cat file

/**
 * @param uri, the file uri to cat 
 * @param cb, callback function. 
 */
cat(uri, cb);

//the callback function
/**
 * @param err, the err message of callback, the result param will be null if error, please check the err before using the result information.
 * @param result, json object including the stream, size, name information. example {stream: readableStream, size: 1024, name: "testfile"}
 */
 cb(err, result)

write

write file to file system

/**
 * @param path, string, the path 
 * @param data, the input data, 'stream.Readable' or 'Buffer'
 * @param cb, the callback function
 */
write(path,data,cb);

//the callback function
/**
 * @param err, the err message of callback, the result param will be null if error, please check the err before using the result information.
 * @param result, string message, example, "write success"
 */
cb(err, result);

Union Folders API

This is a unique interface which can serve content from several providers. File systems are listed as named folders in the root directory.

For example, to setup a union file system testing several folders providers:

npm install folders
npm install folders-ftp
npm install folders-ssh

var mounts = [
	{ "stub" : fio.provider("stub") },
	{ "local" : fio.provider("local") },
	{ "memory" : fio.provider("memory") },
	{ "ftp" : fio.provider("ftp", {
                connectionString : "ftp://test:123456@localhost:3333",
                enableEmbeddedServer : true
        }) },
        { "ssh" : fio.provider("ssh", {
                connectionString : "ssh://test:123456@localhost:3334",
                enableEmbeddedServer : true
        }) }
];

var Fio = require('folders');
var unionfs = new ((Fio.union())(fio, mounts, {
	"view" : "list"
}));

unionfs.ls('.', function(data) {
	// will list the five modules listed as root folders.
});

Sync Union Folders API

A special Union Folders with two mounts, A source folders and destination folders.

Support ls,sync feature.

• ls: will compare two child systems and show the files only in source folders. we support custom LS Handler, Options.logicHandler.

• sync: will sync the files only in source folders to destination folders.

• scheduleSync: crontab sync.

setup a sync Union file system.

var FoldersSyncUnion = function(mounts, options, prefix){...};

• param mounts, the source/dest provider informations.

//example to sync file from /S3/us-east-1/foldersio to root of HDSF folders
var awsConfig = {
  "accessKeyId" : "=== AWS KEY ===",
  "secretAccessKey" : "=== AWS ACCESS KEY ===",
  "service" : "S3",
  "region" : "us-east-1",
  "bucket" : "foldersio",
  "partSize" : 10485760,
  "queueSize" : 5
};

var hdfsConfig = {
  baseurl : "=== WEBHDFS URL ===",
  username : 'hdfs'
};

var mounts = {
  source : {
    module : 'aws',
    opts : awsConfig,
    dir : '/S3/us-east-1/foldersio'
  },
  destination : {
    module : 'hdfs',
    opts : hdfsConfig,
    dir : '/'
  }
};

• param options, Sync Options:

{
  // filter: the filename regex filter, the regex for filter the file in source/dest,
  // Default to null which not filter any file
  filter : '*.txt',

  // if Ignore case of file name when compare file
  ignoreCase : false,

  // if Compare size when compare file
  compareSize : false,
  
  // if compare the whole relative file path (include dir path) or just the file name
  ignoreDirPath : false 

  // number of maximum concurrent transfer threads, copy file.
  concurrency : 5,

  // Compare logic handler, may support different custom logic functions for LS/Cat ...
  // by default, we do subtraction which meaning only show the file in source but not in dest.
  logicHandler: resultFolders = function(sourceFolders, destFolders, options);
}

Call the Sync method

var Fio = require('../src/api');
var SyncUnion = Fio.syncUnion();
var syncUnion = new SyncUnion(mounts, syncOptions);
syncUnion.sync(function(err, result) {
  if (err) {
    return console.log('union sync error: ', err);
  }
  console.log('union sync success, ', result);
});

Crontab sync

// crontab Example, execute sycn every minute.
var jobId = syncUnion.scheduleSync("*/1 * * * *");