Skip to content

Collection of RDF-Connect Typescript processor to deal with file operations

License

Notifications You must be signed in to change notification settings

rdf-connect/file-utils-processors-ts

Repository files navigation

file-utils-processors-ts

Bun CI npm

RDF-Connect Typescript processors for handling file operations. It currently exposes 6 functions:

This function relies on the glob library to select a set of files according to a shell expression and stream them out in a sequential fashion. A wait parameter can be defined to wait x milliseconds between file streaming operations.

This function reads all the files present in a given folder and streams out their content in a sequential fashion. A maxMemory parameter can be given (in GB) to defined threshold of maximum used memory by the streaming process. When the threshold is exceeded, the streaming process will pause for as many milliseconds as defined by the pause parameter.

This function transform a stream by applying a given string substitution on each of the messages. The matching string can be a regex defined by the source property and setting the regexp property to true.

This function substitute all the defined environment variables on each of the elements of an input stream that have been labeled with a ${VAR_NAME} pattern.

This function can read on demand and push downstream the contents of a file located in a predefined folder. This processor is used mostly for testing and demonstrating pipeline implementations.

This function can receive a zipped file in the form of a Buffer and stream out its decompressed contents.

About

Collection of RDF-Connect Typescript processor to deal with file operations

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published