SDA-Filesystem / Data Gateway

This project has been rebranded as Data Gateway

Data Gateway makes use of the:

• SD Connect Proxy API
• SD Apply/SD Submit Download API

It builds a FUSE (Filesystem in Userspace) layer and uses Airlock to export files to SD Connect. Software currently supports Linux, macOS and Windows for:

• Graphical User Interface
• Command Line Interface

Binaries are built on each release for all supported Operating Systems.

Requirements

Go version 1.21

Set these environment variables before running the application:

• FS_SD_CONNECT_API - API for SD-Connect
• FS_SD_SUBMIT_API – a comma-separated list of APIs for SD Apply/SD Submit
• SDS_ACCESS_TOKEN - a JWT for authenticating to the SD APIs
• FS_CERTS - path to a file that contains certificates required by SD Connect, SD Apply/SD Submit, and SDS AAI

Optional envronment variables:

• CSC_USERNAME - username for SDA-Filesystem
• CSC_PASSWORD - password for SDA-Filesystem and Airlock CLI

For test environment follow instructions at https://gitlab.ci.csc.fi/sds-dev/local-proxy

Graphical User Interface

Dependencies

cgofuse and its dependencies on different operating systems.

Install Wails and its dependencies.

Install pnpm

Build and Run

Before running/building the repository for the first time, generate the frontend assests by running:

pnpm install --prefix frontend
pnpm --prefix frontend run build

To run in development mode:

cd cmd/gui
wails dev

To build for production:

cd cmd/gui

# For Linux and macOS
wails build -upx -trimpath -clean -s

# For Windows
wails build -upx -trimpath -clean -s -webview2=embed

Deploy

See Linux setup.

Command Line Interface

Two command line binaries are released, one for SDA-Filesystem and one for Airlock.

SDA-Fileystem

The CLI binary will require a username and password for accessing the SD-Connect Proxy API. Username is given as input. Password is either given as input or in an environmental variable.

Build and Run

go build -o ./go-fuse ./cmd/fuse/main.go

Test install.

./go-fuse -help                        
Usage of ./go-fuse:
  -alsologtostderr
    	log to standard error as well as files
  -http_timeout int
    	Number of seconds to wait before timing out an HTTP request (default 20)
  -log_backtrace_at value
    	when logging hits line file:N, emit a stack trace
  -log_dir string
    	If non-empty, write log files in this directory
  -loglevel string
    	Logging level. Possible values: {debug,info,warning,error} (default "info")
  -logtostderr
    	log to standard error instead of files
  -mount string
    	Path to Data Gateway mount point
  -project string
    	SD Connect project if it differs from that in the VM
  -sdapply
      Connect only to SD Apply
  -stderrthreshold value
    	logs at or above this threshold go to stderr
  -v value
    	log level for V logs
  -vmodule value
    	comma-separated list of pattern=N settings for file-filtered logging

Example run: ./go-fuse -mount=$HOME/ExampleMount will create the FUSE layer in the directory $HOME/ExampleMount for both 'SD Connect' and 'SD Apply'.

User input

User can update the filesystem by inputting the command update. This requires that no files inside the filesystem are being used. Update also clears cache. As a result of this operation, new files may be added and some old ones removed.

The filesystem can be also updated programatically with the SIGUSR2 signal.

To update filesystem on bash in SD Desktop:

# Update CLI version
kill -s SIGUSR2 $(pgrep go-fuse)

# Update GUI version
kill -s SIGUSR2 $(pgrep sda-fuse)

If the user wants to update particular SD Connect files inside the filesystem, the user can input command clear <path>. <path> is the path to the file/folder that the user wishes to update. <path> must at least contain a bucket, i.e. SD-Connect/project/bucket or SD-Connect/project/bucket/file would be acceptable paths, but not, e.g., SD-Connect/project. If the user gives a path to a folder, all files inside this folder are updated but no files are added or removed. This operation clears the cache for all the neccessary files so that the new content is read from the database and sizes of these files are updated in the filesystem.

Airlock

The CLI binary will require a username, a bucket and a filename. Password is either given as input or in an environmental variable.

Build and Run

go build -o ./airlock ./cmd/airlock/main.go

Test install.

./airlock -help
Usage of ./airlock:
  -alsologtostderr
    	log to standard error as well as files
  -debug
    	Enable debug prints
  -journal-number string
    	Journal Number/Name specific for Findata uploads
  -log_backtrace_at value
    	when logging hits line file:N, emit a stack trace
  -log_dir string
    	If non-empty, write log files in this directory
  -logtostderr
    	log to standard error instead of files
  -original-file string
    	Filename of original unecrypted file when uploading pre-encrypted file from Findata vm
  -project string
    	SD Connect project if it differs from that in the VM
  -quiet
    	Print only errors
  -segment-size int
    	Maximum size of segments in Mb used to upload data. Valid range is 10-4000. (default 4000)
  -stderrthreshold value
    	logs at or above this threshold go to stderr
  -v value
    	log level for V logs
  -vmodule value
    	comma-separated list of pattern=N settings for file-filtered logging

Example run: ./airlock username ExampleBucket ExampleFile will export file ExampleFile to bucket ExampleBucket.

Troubleshooting

See troubleshooting for fixes to known issues.

License

Data Gateway is released under MIT, see LICENSE.

Wails is released under MIT

CgoFuse is released under MIT