common_file_system

Common File System

Winslow requires a common file system to store resource and workspace files as well as project, pipeline and stage information. The common working directory of winslow is located either at /winslow/ or the directory the JAR is executed at.

A working directory could look like this:

├── logs
│   ├── 0851843b-4fc0-41df-a99c-c61f7716c413.0000_hello-world
│   ├── 0851843b-4fc0-41df-a99c-c61f7716c413.0001_write-to-workspace-file
│   ├── 0851843b-4fc0-41df-a99c-c61f7716c413.0002_print-env-variable
│   ├── 0851843b-4fc0-41df-a99c-c61f7716c413....
│   ├── 0ad75f6f-2adb-445d-a6be-9233ad4e6f59....
│   ├── ...
├── nomad
│   ├── 0851843b-4fc0-41df-a99c-c61f7716c413.pipeline.toml
│   ├── 0ad75f6f-2adb-445d-a6be-9233ad4e6f59.pipeline.toml
│   ├── ...
├── pipelines
│   ├── pipeline1.toml
│   ├── pipeline2.toml
│   ├── ...
├── projects
│   ├── 0851843b-4fc0-41df-a99c-c61f7716c413.toml
│   ├── 0ad75f6f-2adb-445d-a6be-9233ad4e6f59.toml
│   ├── ...
├── resources
│   ├── resource1.file
│   ├── resource2.file
│   ├── ...
├── run
│   ├── events
│   ├── nodes
│   └── nomad
└── workspaces
    ├── 0851843b-4fc0-41df-a99c-c61f7716c413
    │   ├── 0000_hello-world
    │   │   └── file1
    │   ├── 0001_write-to-workspace-file
    │   │   ├── file1
    │   │   ├── file2
    │   ├── 0002_print-env-variable
    │   │   ├── file1
    │   │   ├── file2
    │   ├── ...
    ├── 0ad75f6f-2adb-445d-a6be-9233ad4e6f59
    ├── ...

Explanation

• logs: Location of logs of all stages ever executed
• nomad: In case of using nomad as Orchestration Backend, this is its working directory
• pipelines: Directory where all pipeline definitions are located in
• projects: Directory where all project definitions are located in
• resources: Global resource directory accessible for each stage
• run: Directory holding volatile but for the operation required information. It is recommended to mount this directory as tmpfs because of its high write activity (spare your SSD)
• workspaces: In this directory, each project has its own sub-directory. Within each such sub-directory, a directory for every executed stage is created. Each new stage inherits the files of the previous stage.

Requirements & Pitfalls

Requirement: For Winslow to communicate correctly with other instances, the common file system must provide atomic file move and atomic file create operations.

To be able to spin up new container instances (stages), Winslow must know how to connect a container to the common file system. This information is passed to Winslow by setting WINSLOW_STORAGE_TYPE and WINSLOW_STORAGE_PATH.