st2 pack install/register is not usable in a HA setup

[pietervogelaar]

I have StackStorm running with a custom Puppet module in a High Available setup. Two identical nodes with all API's, packs etc.

However the st2 pack command is completely useless for this setup. What the command does is register some settings in the database and installs the pack on the disk of one of the nodes. And not even the node on which the command is run, but can happen on the OTHER node. Completely weird behaviour.

At the moment I have the following steps to workaround this:

vcsrepo { "${stackstorm_pack_dir}/${pack_name}":
  ensure   => $pack_ensure,
  provider => $pack_source_type,
  source   => "${pack_source}",
  revision => $pack_source_repo_revision,
  user     => $pack_directory_owner,
  group    => $pack_directory_group,
  notify   => Exec["st2 pack register ${pack_name}"],
}

exec { "st2 pack register ${pack_name}":
  path        => '/bin',
  command     => "st2 pack register ${pack_name} --api-key ${api_key}",
  logoutput   => on_failure,
  refreshonly => true,
  notify      => Exec["create virtualenv for pack ${pack_name}"],
}

exec { "create virtualenv for pack ${pack_name}":
  path        => '/bin',
  command     => "${home_dir}/st2/bin/virtualenv -p ${home_dir}/st2/bin/python ${home_dir}/virtualenvs/${pack_name}",
  refreshonly => true,
  unless      => "test -d ${home_dir}/virtualenvs/${pack_name}",
}

# TODO: PIP install if a requirements file exists

It would be ideal if Puppet could run st2 pack install on each node and that each node would contain all the files that way.

[Kami]

Current recommend approach for multi node / HA setup is to do the following on each server where you want to run actions and sensors (aka where you want to deploy the packs):

1. Clone your repo where you version control all the packs you use into /opt/stackstorm/packs. Or clone each individual pack repo into /opt/stackstorm/packs if you don't version control the packs you use yourself.
2. Run sudo st2ctl reload--register-setup-virtualenvs command. This will create Python virtualenvs and install Python dependencies for all the packs.
3. Run sudo st2ctl reload --register-all command. This will register all the resources in the database.

[pietervogelaar]

Does "sudo st2ctl --register-setup-virtualenvs" also do a st2 pack register? That does not seems the case if I read st2ctl --help => "Create Python virtual environments for all the registered packs."

I assume it also doesn't install pip dependencies?

[Kami]

@pietervogelaar Sorry, forgot that step - you also need to run --register-all so final command is sudo st2ctl --register-setup-virtualenvs --register-all or you can run it as two separate commands.

And --register-setup-virtualenvs does install Python dependencies for all packs - it creates virtual environment for each pack and installs the dependencies.

All those commands are also designed to be idempotent so you can run them every time as part of your "deploy pack(s) to servers" flow.

Will also update the comment above to reflect that.

[pietervogelaar]

Okay, thanks for your help! I will change it and implement it this way. I think it would be valuable to make more notice of this in the documentation.

[pietervogelaar]

One other question, at the moment I have the following code in Puppet to remove a pack:

exec { "st2 pack remove ${pack_name}":
  path => '/bin',
  command => "st2 pack remove ${pack_name} --api-key ${api_key}",
}

vcsrepo { "${stackstorm_pack_dir}/${pack_name}":
  ensure  => $pack_ensure,
  require => Exec["st2 pack remove ${pack_name}"],
}

file { "${home_dir}/virtualenvs/${pack_name}":
  ensure  => $pack_ensure,
  require => Exec["st2 pack remove ${pack_name}"],
}

Is there a better way?

[Kami]

Okay, thanks for your help! I will change it and implement it this way. I think it would be valuable to make more notice of this in the documentation.

You are welcome.

And yes, I will make sure this gets documented :)

As far as removal goes - right now that's the best way to do it (you do need to run pack remove to make sure content gets removed / unregistered from the database).

[pietervogelaar]

One addition, the command is sudo st2ctl reload --register-setup-virtualenvs --register-all.

The reload subcommand was missing.

[Kami]

Yeah, sorry - will update the comment above and add missing "reload".

[pietervogelaar]

I discovered that when a rule, trigger, action or workflow is removed from a git repository and is no longer on the disk of a stackstorm server, the command sudo st2ctl reload --register-setup-virtualenvs --register-all does not purge these items from StackStorm. I have to manually delete them with st2 commands.

It would be very nice if there was a purge flag or something. Is there something available at the moment for this?

[LindsayHill]

No, I don't think there are any out of the box commands for this, apart from st2 pack remove if you're removing the whole pack. Otherwise you have to manually remove each action/rule with commands like st2 action delete, etc.

[pietervogelaar]

For now I will go with st2 pack remove, but this obviously has the downside that actions are not available for a short time. I created an issue for this #3313

[Kami]

I think one a good mid-term solution would be to add "--purge" or similar flag to st2-register-content.

This way it can be used in distributed / HA environment with the existing --register-setup-virtualenvs flag.

What do other people think?

[pietervogelaar]

I found out that the "st2 pack remove" workaround is not really an option. It removes the pack from the database but also from disk. So the code below will result in a removed pack instead of a replaced pack:

    vcsrepo { "${stackstorm_pack_dir}/${pack_name}":
      ensure   => $pack_ensure,
      provider => $pack_source_type,
      source   => $pack_source,
      revision => $pack_source_repo_revision,
      user     => $pack_directory_owner,
      group    => $pack_directory_group,
      notify   => Exec["st2 pack remove ${pack_name}"],
    }

    exec { "st2 pack remove ${pack_name}":
      path        => '/bin',
      command     => "st2 pack remove ${pack_name} --api-key ${api_key}",
      logoutput   => on_failure,
      refreshonly => true,
      notify      => Exec["st2ctl reload ${pack_name}"],
    }

    exec { "st2ctl reload ${pack_name}":
      path        => '/bin',
      command     => 'st2ctl reload --register-setup-virtualenvs --register-all',
      logoutput   => on_failure,
      refreshonly => true,
    }

[estee-tew]

https://stackstorm-community.slack.com/archives/community/p1493064170943011

[stale]

Thanks for contributing to this issue. As it has been 90 days since the last activity, we are automatically marking is as stale. If this issue is not relevant or applicable anymore (problem has been fixed in a new version or similar), please close the issue or let us know so we can close it. On the contrary, if the issue is still relevant, there is nothing you need to do, but if you have any additional details or context which would help us when working on this issue, please include it as a comment to this issue.

[winem]

This is no longer the case.

The new ha-setup is based on kubernetes and the installation of (custom) packs is described well.
https://docs.stackstorm.com/install/k8s_ha.html#custom-st2-packs
https://github.com/stackstorm/stackstorm-ha#install-custom-st2-packs-in-the-cluster

[cognifloyd]

We just released v0.70.0 of our official stackstorm-ha helm chart (to run StackStorm in Kubernetes). This version makes it possible to use st2 pack install with the HA setup by leveraging shared volumes (like NFS) for /opt/stackstorm/packs and /opt/stackstorm/virtualenvs. https://github.com/StackStorm/stackstorm-ha/releases/tag/v0.70.0

There are a few things mentioned in this issue that have not been implemented (like the st2ctl reload ... --purge option), but the primary request in this issue was to make st2 pack install work for HA. As our official HA install supports st2 pack install now, I'm going to close this issue. Other issues or PRs should be filed for other improvements around HA.

Thanks!