User guide
Please note that this documentation is in the process of being refactored and as such may contain missing links or references to sections / pages that have been moved elsewhere.
Have a look at the HOWTO section.
The commands to handle project creation are all under the seed namespace of Hem. At the time of writing, there is only one command available: plant.
When you plant a project seed you must provide two values:
- Repository URL
- Seed name
The repository URL is the URL of your project repository. Hem only supports git.
The seed name is the name or location of the seed you wish to base your project on. If the seed name looks like a local path then Hem will try to clone the seed from that path. If it does not, Hem will prepend 'hem-seed-' to the value you specify and attempt to clone it from the Inviqa github repository by that name.
For example:
hem seed plant my-project
This will interactively prompt you for the two necessary parameters. Providing empty values to these prompts will cause Hem to use the defaults shown in square brackets.
You can also specify these parameters on the commmand line with the --git-url and --seed options.
hem seed plant my-project --git-url=http://some.custom.url/my-project" --seed="./custom-local-seed"
This will attempt to create the project using the seed in the "customer-local-seed" directory located in the current directory.
Once your project seed has been planted, Hem will also make the first commit with all of your project files and configure the project origin in git.
Hem seeds are just git repositories so you can simply create a new git repository and point to it with the "Seed name" parameter when running hem seed plant.
While developing the seed locally you can use the relative or absolute pathname to the seed but you must ensure that you have committed any changes to ensure that they are used when you plant the seed. Hem utilizies git clone and git pull to clone and update a cache of the seeds
During the plant process, Hem will perform the following steps:
- Clone or update the seed repository
- Export the seed repository to your project directory
- Initialize and update any submodules the seed uses
- Export the submmodules to your project directory
- Execute the "seedinit.rb" file contained in the seed if there is one
- Perform placeholder string replacements on the exported files
- Initialize a git repository in the project directory
- Add all files & make an initial commit
- Configure the git remote "origin"
So that the seeds are not re-downloaded every time that you use them they are locally cached to ~/.hem/seeds. This cache is a bare clone so it does not have a working tree. If you wish to inspect the working tree of a local seed cache, the easiest way is to clone the cached folder:
git clone ~/.hem/seeds/default/ hem-seed-default
The above command will clone the cache for the default seed to the "hem-seed-default" folder in the current directory.
The seedinit.rb file is executed during the seed plant process after the seed and any submodules are exported but before placeholder replacements. This file allows seeds to implement custom logic. As an example, the magento seed uses this file to download the magento sample data as part of the seed plantation process.
When Hem plants a project seed, it searches for .erb templates, and translates them using Ruby ERB template engine. It has access to the Hem.project_config via the config variable
The structure of this config by default is as follows:
- name
- git_url
- hostname
- vm_ip
- asset_bucket
- seed
- name
- url
- version
Any of these values can be referenced in files as <%= config.variable_name %>. For nested structures, a dot is used to separate, e.g. <%= config.seed.name %>.
The entire file tree is searched for .erb files and are saved back with the .erb extension removed from the filename.
The hem vm commands have been written to assist with starting, stopping and connecting to your VMs.
The hem vm commands are "project" commands. This means that you must be within a project folder in order to run them. See Project directory detection for more on how this works.
Within a project folder you can start a VM with all dependencies handled by running the following command:
hem vm up
This command will attempt the following:
- Install gems if a Gemfile is present
- Fetch composer.phar if composer.json is present
- Install composer dependencies if composer.json is present
- Install chef cookbooks if a Cheffile is present
- Download project assets (database dump / images etc)
- Start the VM (without provision)
- Provision the VM
- Apply project assets
The project asset tasks will only execute if you have both configured your AWS credentials and your project has been configured with an S3 bucket. See Asset synchonisation for more information.
You can stop a VM from anywhere within your project folder by executing the following command:
hem vm stop
This will suspend the VM. If you wish to destroy the VM...
You can destroy a VM by executing the following command:
hem vm destroy
Sometimes you will need to completely rebuild the VM. You can do this easily with Hem using:
hem vm rebuild
Opens an SSH session on the VM:
hem vm ssh
You can also pipe content through SSH and execute arbitrary commands as with he real SSH command:
echo "This is a test `hostname`" | hem vm ssh -- sed -e 's/$/`hostname`'
Please note that this command piping feature is experimental and may have buggy escaping.
Opens a MySQL CLI session on the VM:
hem mysql console
You can also pipe SQL directly in to this command as if it were mysql:
echo "SHOW DATABASES" | hem mysql console
Hem assets functionality allows projects to store, share and retrieve project related asset files. Asset files are things like database dumps, images and any other files that may need to be shared that are not suitable for Github. Please note that sensitive data should not be stored using the assets mechanism. This includes customer data in database dumps so please ensure that this information is stripped from a dump before it is uploaded.
Hem asset sync utilizes Amazon S3 as a storage and retrieval mechanism via the API. Because of this fact, you will need an account on Inviqa AWS and you will need to configure Hem with your API credentials.
If you need a login, please contact mike@inviqa.com or internalsupport@inviqa.com.
You can configure hem with your credentials with the hem host config command.
Your project will also need to have had an S3 bucket created for it. Please see the S3 access control section for more information.
In order to download project assets from anywhere within the project directory (once your AWS credentials are configured) you can do so with the following command:
hem assets download
This will download all assets to tools/assets/development for the development environment. If you wish to download a different set of assets, you can do so by specifying the --env option.
hem assets download --env=uat
This will download all assets to tools/assets/uat.
Note that the UAT environment must have assets in a folder called 'uat' in the S3 bucket for this to work. Specifying an invalid env will result in an error.
Assets can be uploaded with the following command:
hem assets upload
This will sychronize all assets from tools/assets/development to the 'development' folder in the S3 bucket. If the S3 bucket contains files that are not present in the tools/assets/development folder, they will be removed from S3 without warning.
As with downloading, you can specify the environment for asset uploading by passing the --env option.
hem assets upload --env=uat
This will upload the contents of tools/assets/uat to the 'uat' folder on S3.
You can apply assets with the following command:
hem assets apply
This will scan assets in the tools/assets/development folder for recognized filename patterns and run the relevant applicator for them.
As with the other asset commands, --env can be used to apply assets from another environment. Please note that by default the vm:up task will apply assets from the development environment and there is presently no way to override the environment. As such applying assets from other environments to your development VM is presently not recommended.
hem assets apply --env=uat
Hem expects your project to have its own S3 bucket. Access control is managed using IAM with the following strategy:
- Each project has its own bucket
- Each project has its own group
- The project group grants access only to the project bucket
- Users are added to the project group as required
This scheme means that support can easily delete a user from the AWS account to revoke all asset access. It also allows for client access to assets by means of a client account for AWS. If necessary, clients can be added to a read-only group for the project to prevent any potential mishaps or abuse.
New buckets usually require at least 30 minutes for DNS propagation.
When running hem assets upload against a fresh bucket, you might encounter the following error message:
(AWS::S3::Errors::TemporaryRedirect) Please re-send this request to the specified temporary endpoint. Continue to use the original request endpoint for future requests.This might be caused by regional config distribution. Simply waiting ~1.5-2 hours normally fixes it.
Hem supports synchronisation both from remote to local and local to remote. It does not support remote to remote so if this is required you'll need to download the assets locally first.
The synchronisation mechanism is very simple. All files within the assets folder (tools/assets/development for the development environment) will be md5-summed and the filenames / hashes compared against those remotely. S3 exposes the md5sum of a file as its etag. A delta is then created containing files to add and files to remove. The S3 synchonisation code simply iterates these deltas to download, upload or delete files as necessary.
At the time of writing there are no prompts or checks for deleting asset files. This may change in the future to prevent accidents.
Hem knows how to apply some assets based on some filename hints.
Files called something.anythinghere.sql.gz will be treated as mysql dumps. Everything up to the first '.' will be taken as the database name. Everything else is ignored so you may include version strings or such. Hem blindly applies all asset files matching this patterns so at the time of writing it is not possible to store multiple versions of a database in the assets mechanism.
When applying an SQL dump, Hem will check to see if the database exists. If it does, it is assumed that the file has already been applied and application will be skipped.
Files named something.files.tgz (or .tar.gz or .tar.bz2) will be treated as tarballs containing files to be extracted to the project root. Any file archives should include all directory structure required so, for example, if your archive contains a bunch of images that need to reside at public/uploads/images then this must be the path that they have inside the archive. At the time of writing it is not possible to tell Hem where to extract an archive (they're all extracted to project root). File archives do not have any re-application checks and will be re-extracted every time they are applied.
If you have need of a custom asset applicator (or wish to override the defaults) you can do so by registering your asset applicator with the Hem.asset_applicators.register method in your Hemfile. An example of such follows:
Hem.asset_applicators.register 'zip', /.*\.zip/ do |file|
Dir.chdir Hem.project_path do
shell 'unzip', file.shellescape
end
endThis asset applicator will automatically unzip any file ending with '.zip' in the assets folder when assets are applied.
The deps tasks are simple tasks to install project dependencies from tools like librarian-chef, composer and bundler.
Since they are very thin wrappers around the normal commands to perform such tasks they are hidden in the normal hem command list. However, the commands can still be invoked:
hem deps gems
Of special note is the composer dependency installer. This is because it will first attempt to leverage your host php installation if you have one to install the dependencies. If that fails or PHP is not available on the host it will retry using PHP in the VM.
Executing composer on the host has the benefit of being able to utilize the composer cache across VM rebuilds but requires that the host PHP install be correctly configured for PHP.
The hem tools commands fetch common tools to make them available for the project. At the time of writing, only composer is implemented but n98magerun will be implemented for the hem magento seed.
The hem system tasks are present to provide assistance and additional tooling for host machine configuration.
You can check key aspects of the configuration of your system with the following command:
hem system check
This command will run a suite of tests and highlight any potential problem areas. Advice is given on steps to remedy if a problem is encountered.
As Hem is a cli tool designed to make developers lives easier shell completions are essential. At the time of writing only Fish completions have been implemented. PRs are welcome for your shell of choice, especially Bash or ZSH.
To install fish shell completions:
hem system completions install --fish
This should make completions available immediately. Hem completions are cached based on directory name under /tmp/fish_hem_completion_cache/ because they're generated utilizing the same mechanisms as the help formatter and a small but noticeable delay will be encountered when this happens.
Completions are cached based on directory as each project may have a different set of tasks but the cache is not smart enough to use the same cache for different directories in the project (yet).
Hem needs very little manual configuration but for what information is required this command exists.
At the moment only AWS credentials are configurable here. These are required to plant a magento seed or utilize project asset functionality. If you have the variables set in your ENV for the AWS cli client (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY) then Hem will try to use those if you don't have any specifically configured for Hem.
hem config
This command will interactively prompt you for configuration. Leaving a response blank will use the default shown in square brackets.
Hem will attempt to automatically determine the root of your project folder structure. It does so by attempting to locate some files in common locations. It will traverse up the directory tree searching for these files until it either finds one or hits the root folder of the filesystem.
The files searched for are:
- Hemfile
- Hobofile
- tools/hem
- tools/hobo
- tools/vagrant/Vagrantfile
If your project does not have any of these files, create an empty file called "Hemfile" at the root of your project.
hem seed plant my-super-project
Add a file called 'Hemfile' to the root of your project and add it to git.
Per project configuration is expected to be in tools/hem/config.yaml.
The asset synchronization will look at the 'asset_bucket' key in the root config to determine which S3 bucket to use. This bucket must be created in S3 along with the appropriate group policy and the name of the bucket set in this key. e.g.
---
:asset_bucket: inviqa-assets-redbull
The S3 bucket name should match that of the project for discovery and admin reasons.
By default hem will assume that the VM has been configured to allow passwordless mysql login. If this is not the case, as is likely for 99% of existing projects then you must specify the username / password in tools/hem/config.yaml.
If the username is root it can be omitted as this is the default.
---
:mysql:
:password: my-root-password
Hem was designed with project extensibility in mind. As such it is trivial to add new commands to your project. See the Hemfile DSL page for more detailed information on the DSL in use.
An example task in the project Hemfile:
desc "My project"
namespace :myproject do
desc "My task"
task :mytask do
puts "Do some stuff"
end
end
This task will now appear in the Hem command list and can be invoked as follows:
hem myproject mytask
Top level tasks are also support.
Please note that tasks must have a description to appear in the Hem command list output. If you do not see your command, try hem --all.
Sometimes it is necessary to override built-in Hem behaviour. This is easily accomplished using 'replace' in the Hemfile.
replace 'deps:composer' do
Hem.ui.warning "Didn't want composer deps anyway"
end
This will cause the message to be displayed whenever the "deps:composer" task is invoked. This includes from the command line with hem deps composer or as part of a dependency chain (i.e. during hem vm up).
Adding project specific tasks to existing Hem tasks is trivial...
before 'deps:composer' do
puts 'Yay! Getting some composer deps!'
end
after 'deps:commands' do
puts 'Yay! Got some composer deps!'
end
Unfortunately the capistrano style before 'deps:composer', 'some:other:task' syntax is not supported at this time.
Make sure the files in question are at least staged in the git index; that is to say git status should not show them as 'Untracked files'.
For speed reasons Hem uses git ls-files which means that git must at least know about the files before Hem will locate them.
If Hem is misbehaving or you need more information to diagnose a problem, Hem will happily spew debug information at you with the --debug flag.
hem vm up --debug
Debug output is both sent to terminal and to /tmp/hem_debug.log.
In the event of an error, error details will be stored in /tmp/hem_error.log.
If you report an issue with Hem, we'll probably ask you for either or both of these files.
Docs were dumped in to a single page for ease of editing. Now that the guts of them are here we should split them up.
The basic DSL is documented at Hemfile DSL but supporting libraries are not (S3Sync, HostCheck, Hem::Ui etc).