-
Notifications
You must be signed in to change notification settings - Fork 76
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
11 changed files
with
499 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
# 0.1.0 | ||
|
||
- Initial release |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,180 @@ | ||
# `democratic-csi` CSI plugin | ||
|
||
This pack deploys two jobs that run the | ||
[`democratic-csi`](https://github.com/democratic-csi/democratic-csi) | ||
CSI plugin. The node plugin tasks will be run as a system job, and the | ||
controller tasks will be run as a service job. | ||
|
||
## Client Requirements | ||
|
||
This pack can only be run on Nomad clients that have enabled volumes and | ||
privileged mode for the Docker task driver. In addition, clients will | ||
need to have a source NFS volume mounted to any client host that runs | ||
the controller task. | ||
|
||
### Example NFS Server | ||
|
||
The following is an example of installing and configuring a NFS server | ||
on an apt-based Linux distrobution (ex. Debian or Ubuntu). This | ||
configuration exports the directory `/var/nfs/general` to any Nomad | ||
client on the `192.168.56.0/24` address space (this is commonly used | ||
for Vagrant hosts on Virtualbox). | ||
|
||
```sh | ||
sudo apt-get install nfs-kernel-server | ||
sudo mkdir /var/nfs/general | ||
|
||
sudo cat <<EOF > /etc/exports | ||
/var/nfs/general 192.168.56.0/24(rw,sync,no_subtree_check,no_root_squash) | ||
EOF | ||
|
||
sudo systemctl enable nfs-kernel-server | ||
sudo systemctl start nfs-kernel-server | ||
``` | ||
|
||
The `democratic-csi` controller is unusual in needing to mount the NFS | ||
volume because NFS doesn't have a remote API other than filesystem | ||
operations. So this plugin has to have the NFS export bind-mounted | ||
into the plugin container. Any client running the controller will need | ||
to have the NFS export in the host's `/etc/fstab`. The following | ||
configuration is an example for the NFS export shown above, assuming | ||
that the NFS server can be found at IP `192.168.56.60`. The mount | ||
point `/srv/nfs_data` shown here should be used for the | ||
`nfs_controller_mount_path` variable. | ||
|
||
``` | ||
sudo cat <<EOF >> /etc/fstab | ||
192.168.56.60:/var/nfs/general /srv/nfs_data nfs4 rw,relatime 0 0 | ||
EOF | ||
``` | ||
|
||
## Variables | ||
|
||
The following variables are required: | ||
|
||
* `nfs_share_host` - The IP address of the host for the NFS share. | ||
* `nfs_share_base_path` - The base directory exported from the NFS | ||
share host. | ||
* `nfs_controller_mount_path` - The path where the NFS mount is | ||
mounted as a host volume for the controller plugin. | ||
|
||
For example, using the NFS configuration described above: | ||
|
||
```sh | ||
nomad-pack plan \ | ||
-var nfs_share_host=192.168.56.60 \ | ||
-var nfs_share_base_path=/var/nfs/general \ | ||
-var nfs_controller_mount_path=/srv/nfs_data \ | ||
. | ||
``` | ||
|
||
The following variables are optional: | ||
|
||
* `job_name` (string "democratic_csi") - The prefix to use as the job | ||
name for the plugins. For exmaple, if `job_name = "democratic_csi"`, | ||
the plugin job will be named `democratic_csi_controller`. | ||
* `datacenters` (list(string) ["dc1"]) - A list of datacenters in the | ||
region which are eligible for task placement. | ||
* `region` (string "global") - The region where the job should be | ||
placed. | ||
* `plugin_id` (string "org.democratic-csi.nfs") - The ID to register | ||
in Nomad for the plugin. | ||
* `plugin_namespace` (string "default") - The namespace for the plugin | ||
job. | ||
* `plugin_image` (string | ||
"docker.io/democraticcsi/democratic-csi:latest") - The container | ||
image for `democratic-csi`. | ||
* `plugin_csi_spec_version` (string "1.5.0") - The CSI spec version | ||
that democratic-csi will comply with. | ||
* `plugin_log_level` (string "debug") - The log level for the plugin. | ||
* `nfs_dir_permissions_mode` (string "0777") - The unix file | ||
permissions mode for the created volumes. | ||
* `nfs_dir_permissions_user` (string "root") - The unix user that owns | ||
the created volumes. | ||
* `nfs_dir_permissions_group` (string "root") - The unix group that | ||
owns the created volumes. | ||
* `controller_count` (number 2) - The number of controller instances | ||
to be deployed (at least 2 recommended). | ||
* `volume_id` (string "myvolume") - ID for the example volume spec to | ||
output. | ||
* `volume_namespace` (string "default") - Namespace for the example | ||
volume spec to output. | ||
|
||
#### `constraints` List of Objects | ||
|
||
[Nomad job specification | ||
constraints](https://www.nomadproject.io/docs/job-specification/constraint) | ||
allow restricting the set of eligible nodes on which the tasks will | ||
run. This pack automatically configures the following required | ||
constraints: | ||
|
||
* Plugin tasks will run on Linux hosts only | ||
* Plugin tasks will run on hosts with the Docker driver's | ||
[`volumes`](https://www.nomadproject.io/docs/drivers/docker#volumes-1) | ||
enabled and | ||
[`allow_privileged`](https://www.nomadproject.io/docs/drivers/docker#allow_privileged) | ||
set to `true`. | ||
* The controller plugin tasks will be deployed on distinct hosts. | ||
|
||
You can set additional constraints with the `constraints` variable, | ||
which takes a list of objects with the following fields: | ||
|
||
* `attribute` (string) - Specifies the name or reference of the | ||
attribute to examine for the constraint. | ||
* `operator` (string) - Specifies the comparison operator. The | ||
ordering is compared lexically. | ||
* `value` (string) - Specifies the value to compare the attribute | ||
against using the specified operation. | ||
|
||
Below is also an example of how to pass `constraints` to the CLI with | ||
with the `-var` argument. | ||
|
||
```bash | ||
nomad-pack run -var 'constraints=[{"attribute":"$${meta.my_custom_value}","operator":">","value":"3"}]' packs/democratic_csi_nfs | ||
``` | ||
|
||
#### `resources` Object | ||
|
||
* `cpu` (number 500) - Specifies the CPU required to run this task in | ||
MHz. | ||
* `memory` (number 256) - Specifies the memory required in MB. | ||
|
||
## Volume creation | ||
|
||
This pack outputs an example volume specification based on the plugin variables. | ||
|
||
#### **`volume.hcl`** | ||
|
||
```hcl | ||
type = "csi" | ||
id = "my_volume" | ||
namespace = "default" | ||
name = "my_volume" | ||
plugin_id = "org.democratic-csi.nfs" | ||
capability { | ||
access_mode = "multi-node-multi-writer" | ||
attachment_mode = "file-system" | ||
} | ||
capability { | ||
access_mode = "single-node-writer" | ||
attachment_mode = "file-system" | ||
} | ||
capability { | ||
access_mode = "single-node-reader-only" | ||
attachment_mode = "file-system" | ||
} | ||
mount_options { | ||
mount_flags = ["noatime"] | ||
} | ||
``` | ||
|
||
Create this volume with the following command: | ||
|
||
```sh | ||
nomad volume create volume.hcl | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,11 @@ | ||
app { | ||
url = "https://github.com/democratic-csi/democratic-csi" | ||
author = "Tim Gross <tgross@hashicorp.com>" | ||
} | ||
|
||
pack { | ||
name = "democratic_csi_nfs" | ||
description = "This pack deploys the democratic-csi plugin, configured for use with NFS" | ||
url = "https://github.com/hashicorp/nomad-pack-community-registry/democratic_csi_nfs" | ||
version = "0.1.0" | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,24 @@ | ||
type = "csi" | ||
id = "[[ .my.volume_id ]]" | ||
namespace = "[[ .my.volume_namespace ]]" | ||
name = "[[ .my.volume_id ]]" | ||
plugin_id = "[[ .my.plugin_id ]]" | ||
|
||
capability { | ||
access_mode = "multi-node-multi-writer" | ||
attachment_mode = "file-system" | ||
} | ||
|
||
capability { | ||
access_mode = "single-node-writer" | ||
attachment_mode = "file-system" | ||
} | ||
|
||
capability { | ||
access_mode = "single-node-reader-only" | ||
attachment_mode = "file-system" | ||
} | ||
|
||
mount_options { | ||
mount_flags = ["noatime"] | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,24 @@ | ||
[[- define "constraints" -]] | ||
constraint { | ||
attribute = "${attr.kernel.name}" | ||
value = "linux" | ||
} | ||
|
||
constraint { | ||
attribute = "${attr.driver.docker.privileged.enabled}" | ||
value = true | ||
} | ||
|
||
[[ if .my.constraints ]][[ range $idx, $constraint := .my.constraints ]] | ||
constraint { | ||
attribute = [[ $constraint.attribute | quote ]] | ||
[[- if $constraint.value ]] | ||
value = [[ $constraint.value | quote ]] | ||
[[- end ]] | ||
[[- if $constraint.operator ]] | ||
operator = [[ $constraint.operator | quote ]] | ||
[[- end ]] | ||
} | ||
[[- end ]][[- end ]] | ||
|
||
[[- end -]] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
[[- define "location" -]] | ||
namespace = "[[ .my.plugin_namespace ]]" | ||
region = "[[ .my.region ]]" | ||
datacenters = [[ .my.datacenters | toPrettyJson ]] | ||
[[- end -]] |
19 changes: 19 additions & 0 deletions
19
packs/democratic_csi_nfs/templates/_plugin_config_file.tpl
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,19 @@ | ||
[[- define "plugin_config_file" -]] | ||
|
||
template { | ||
destination = "${NOMAD_TASK_DIR}/driver-config-file.yaml" | ||
data = <<EOH | ||
driver: nfs-client | ||
instance_id: | ||
nfs: | ||
shareHost: [[ .my.nfs_share_host ]] | ||
shareBasePath: "[[ .my.nfs_share_base_path ]]" | ||
controllerBasePath: "/storage" | ||
dirPermissionsMode: "[[ .my.nfs_dir_permissions_mode ]]" | ||
dirPermissionsUser: [[ .mynfs_dir_permissions_user ]] | ||
dirPermissionsGroup: [[ .mynfs_dir_permissions_group ]] | ||
EOH | ||
} | ||
|
||
[[- end -]] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
[[- define "resources" -]] | ||
resources { | ||
cpu = [[ .democratic_csi_nfs.resources.cpu ]] | ||
memory = [[ .my.resources.memory ]] | ||
} | ||
[[- end -]] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,56 @@ | ||
job "[[ .my.job_name ]]_controller" { | ||
[[ template "location" ]] | ||
group "controller" { | ||
count = [[ .my.controller_count ]] | ||
[[ template "constraints" ]] | ||
constraint { | ||
operator = "distinct_hosts" | ||
value = "true" | ||
} | ||
|
||
task "plugin" { | ||
driver = "docker" | ||
config { | ||
image = "[[ .my.plugin_image ]]" | ||
args = [ | ||
"--csi-version=[[ .my.plugin_csi_spec_version ]]", | ||
"--csi-name=[[ .my.plugin_id ]]", | ||
"--driver-config-file=${NOMAD_TASK_DIR}/driver-config-file.yaml", | ||
"--log-level=[[ .my.plugin_log_level ]]", | ||
"--csi-mode=controller", | ||
"--server-socket=${CSI_ENDPOINT}", | ||
] | ||
# normally not required for controller plugins, but NFS | ||
# doesn't have a remote API other than mounting, so this | ||
# plugin has to be able to mount the NFS volume as a | ||
# bind-mount in order to create and snapshot volumes. | ||
privileged = true | ||
mount { | ||
type = "bind" | ||
source = "[[ .my.nfs_controller_mount_path ]]" | ||
target = "/storage" | ||
readonly = false | ||
} | ||
} | ||
[[ template "plugin_config_file" ]] | ||
[[ template "resources" ]] | ||
csi_plugin { | ||
id = "[[ .my.plugin_id ]]" | ||
type = "controller" | ||
mount_dir = "/csi" | ||
} | ||
} | ||
} | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,51 @@ | ||
job "[[ .my.job_name ]]_node" { | ||
# you can run node plugins as service jobs as well, but this ensures | ||
# that all nodes in the DC have a copy. | ||
type = "system" | ||
[[ template "location" ]] | ||
group "node" { | ||
[[ template "constraints" ]] | ||
task "plugin" { | ||
driver = "docker" | ||
env { | ||
CSI_NODE_ID = "${attr.unique.hostname}" | ||
} | ||
|
||
config { | ||
image = "[[ .my.plugin_image ]]" | ||
args = [ | ||
"--csi-version=[[ .my.plugin_csi_spec_version ]]", | ||
"--csi-name=[[ .my.plugin_id ]]", | ||
"--driver-config-file=${NOMAD_TASK_DIR}/driver-config-file.yaml", | ||
"--log-level=[[ .my.plugin_log_level ]]", | ||
"--csi-mode=node", | ||
"--server-socket=${CSI_ENDPOINT}", | ||
] | ||
# node plugins must run as privileged jobs because they | ||
# mount disks to the host | ||
privileged = true | ||
ipc_mode = "host" | ||
network_mode = "host" | ||
} | ||
|
||
[[ template "plugin_config_file" ]] | ||
|
||
[[ template "resources" ]] | ||
|
||
csi_plugin { | ||
id = "[[ .my.plugin_id ]]" | ||
type = "node" | ||
mount_dir = "/csi" | ||
} | ||
|
||
} | ||
} | ||
} |
Oops, something went wrong.