New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Documentation of Configuration Management #855

Open
Drupaldise opened this Issue Apr 8, 2015 · 22 comments

Comments

Projects
None yet
@Drupaldise

Drupaldise commented Apr 8, 2015

Hi, where can I find some docs about the configuration management?
How can i work with the staging dir in files/config/staging - or what is this?
What is the best practice to work with GIT (or other version control) with my exported config files?
In other words how can I deploy my config changes without uploading a file - instead using the filesystem?

Thanks for enlighten me. ;-)

@quicksketch

This comment has been minimized.

Member

quicksketch commented Apr 10, 2015

Hi @Drupaldise! We have an article coming out in Drupal Watchdog next month that includes some details, but I think generally speaking we don't have documentation on managing config via the file system. We need to figure out even where this documentation would go between the API docs and the Backdrop handbook, but generally I think I can sum it up. Perhaps once we have a place to put it, we can get something official written up.


If you're going to be managing configuration outside the user interface (such as through version control or even FTP), you should move the config directories outside of the Backdrop "files" directory. Backdrop by default will put the configuration files in a randomly generated /files/config_[hash] directory, and points settings.php at this path like this:

$config_directories['active'] = 'files/config_b7c6057f5199d7aa921445a3d31d30df/active';
$config_directories['staging'] = 'files/config_b7c6057f5199d7aa921445a3d31d30df/staging';

Because the "files" directory is generally not in version control, you can make managing your config easier (and more secure) if this directory is moved outside the web directory. Generally that means up one level in the file system:

$config_directories['active'] = '../config/active';
$config_directories['staging'] = '../config/staging';

And you'll need to organize your Git repository or site directory to match this structure, so it would be something like:

  • .git
  • .gitignore
  • config
    • active
    • staging
    • versioned (more on this later)
  • docroot
    • .htaccess
    • core
    • files
    • layouts
    • modules
    • ... etc. (the Backdrop installation).

I included the .gitignore file in here because you'll probably want to make a few entries in this file. I'd suggest ignoring both the "files" directory and the active and staging config directories individually:

/config/active
/config/staging
/docroot/files

The key to all management of config is the "staging" directory. This directory will be empty all the time unless you're actively moving configuration. We're also introducing our own "versioned" directory that we'll manage in Git.

On your localhost or development machine. Configure your site as you would want to deploy to live. Then to deploy your changes simply copy the "active" files into the "versioned" directory, then check it into Git.

cp config/active/* config/versioned/
git add versioned -A
git commit -m "Updating config files."
git push origin master

Then on your live server, copy the files back out of "versioned" directory into the "staging" (NOT the "active") directory:

ssh me@example.com
git pull origin master
cp config/versioned/* config/staging/

You've now "staged" your configuration for deployment, but it's not yet active. You have to let Backdrop import the configuration to allow it to make necessary database changes and validate the configuration is acceptable (e.g. won't accidentally delete in-use fields).

So visit admin/config/development/configuration in your Backdrop site, where you should now see the staged changes. You can view the differences in each file that was modified, deleted, or added. Click the "Import all" button. If no validation errors occurred, the config will be imported and active. Now all menus, content types, theme settings, etc. that you configured on your localhost will be reflected on the live site.

In the future, we'll probably have a command-line tool (#47) to do this last step without the UI, but it's a good idea to use the UI to view the file diffs anyway.

Note that after you do the import, your "staged" directory will be empty again. Ready for the next time you do a deployment and copy from the "versioned" directory. If you would like to pull down the configuration from live, you can simply do the same process of copying the active directory to versioned, checking it in, pulling down on your local, copying to staging, then importing.


There may be other approaches that work as well. We've theorized ideas like managing the "staging" directory all the time, or a combination of symlinks, but so far I think the "just make another directory" for versioned config is the easiest approach we've encountered.

@jenlampton

This comment has been minimized.

Member

jenlampton commented Apr 10, 2015

@quicksketch That's an interesting approach. I'm going to outline this approach below, along with the two others, in more detail for a better compare / contrast for anyone curious.

The extra directory approach

Configuration of config directory

config/
config/active
config/staging
config/versioned <-- track via Git

Additional set up

  1. Move config outside of files (and preferably outside the docroot)
  2. Update settings.php to point at the new locations of active & staging

Deploying to prod - moving upstream

  1. local: overwrite versioned directory with active (deletions too!)
  2. local: add, commit, push via Git
  3. production: pull changes
  4. production: copy files
  5. production: run config importer (via UI)

Sync local from prod - moving downstream

  1. production: overwrite versioned directory with active (deletions too!)
  2. production: add, commit, push via Git
  3. local: pull changes
  4. local: copy files
  5. local: run config importer (via UI)

Upsides:

  1. All files tracked via Git will remain "up to date" after importing staged config

Downsides:

  1. Accidental reverts are possible - see (1) below

There are two other approaches we've also been considering:

The versioned staging directory approach

This an idea from @populist and @davidstrauss based on how the config directory is being managed for Drupal 8 on Pantheon.

Configuration of config directory

config/
config/active
config/staging <-- track via Git

Additional set up

  1. Move config outside of files (and preferably outside the docroot)
  2. Update settings.php to point at the new locations of active & staging

Deploying to prod - moving upstream

  1. local: overwrite staging directory with active (deletions too!)
  2. local: add, commit, push via Git
  3. production: pull changes
  4. production: run config importer (via UI)

Sync local from prod - moving downstream

  1. production: overwrite staging directory with active (deletions too!)
  2. production: add, commit, push via Git
  3. local: pull changes
  4. local: run config importer (via UI)

Upsides:

  1. No extra directories needed
  2. One less step (than above) when moving upstream & downstream
  3. When deployed, the config changes would "land" in your staging directory on production, ready to be imported.

Downsides:

  1. Accidental reverts are possible - see (1) below
  2. Git reports staging directory as always "out of date" - see (2) below

And this approach is my personal favorite (and what we'll be recommending in the Drupal Watchdog article, out this May):

The figure-8 approach

This approach was recommended by @heyrocker after creating CMI for Drupal 8. It involves using your settings.php file to define different active and staging directories for each environment.

Configuration of config directory

config/
config/dev-active <-- track via Git
config/prod-active <-- track via Git

Additional set up

  1. Move config outside of files (and preferably outside the docroot)
  2. Rename config/active to something like config/prod-active
  3. rename config/staging to something like config/dev-active.
  4. On your local site, update settings.php to point at dev-active for active, and prod-active for staging.
  5. On your production site, update settings.php to point at prod-active for active, and dev-active for staging.

Deploying to prod - moving upstream

  1. local: add & commit all files in config exactly as they are. push.
  2. production: pull changes
  3. production: run config importer (via UI)

When you pull upstream, the config files from what was the active directory on local land in what is now the staging directory on production - ready for import.

Sync local from prod - moving downstream

  1. production: add & commit all files in config exactly as they are. push.
  2. local: pull changes
  3. local: run config importer (via UI)

When you pull downstream, the config files from what was the active directory on production land in what is now the staging directory on local - ready for import.

Upsides:

  1. Once set up, you never need to think about which directory is which!
  2. No extra directories needed
  3. One more less step to move both upstream & downstream
  4. Commit things as they are, no extra moving files around.

Downsides:

  1. Git reports the (staging) directory as always "out of date" - see (2) below
  2. Figuring this out may make your head explode

Here is a more detailed explanation of the two downsides mentioned above:

(1) ACCIDENTAL REVERTS ARE POSSIBLE: If there was ever a config change made on production and not actively copied out of the active directory and committed wherever it's supposed to be tracked, the next time code is deployed that production change would be reverted.

(2) GIT SHOWS STATUS: OUT OF DATE. Git is tracking and reporting all the deleted config files in the staging directory. Running a Git status check on any environment will show the deleted files. Developers may get used to seeing deleted files in staging, and train themselves to ignore those warnings. When a config file is actually deleted BY MISTAKE this is very likely to be overlooked!

@Drupaldise

This comment has been minimized.

Drupaldise commented Apr 10, 2015

Hey Guys, thanks for pointing this out. It makes it more clear for me now. In my opinion there must be a topic for that on backdropcms.org Handbook. This feature is so important.

@heyrocker

This comment has been minimized.

heyrocker commented Apr 10, 2015

Note that my 'figure 8' idea stems from the very early stages of CMI, and I don't think it translates well to the way that the modern system (in Drupal or Backdrop) is architected and the thought-work that has gone into it since then. I hadn't considered the third directory option. It's a solid solution but I wish we had something a little more seamless. The whole thing of git status being out of date never really bothered me.

@jenlampton

This comment has been minimized.

Member

jenlampton commented Apr 10, 2015

@heyrocker what is it about the figure-8 that doesn't translate well to how Drupal 8 & Backdrop are architected? What is it about the other two options that translates better? I still prefer the figure-8 approach since it removes the need for copying anything anywhere (and needing to carefully think about where to copy it). Also, this set-up lends itself nicely to having more than 2 environments (like on Pantheon or Acquia) just by adding additional directories.

Config Local Dev Test Live
jen-active deploy
dev-active deploy
test-active deploy
live-active deploy
@mikemccaffrey

This comment has been minimized.

mikemccaffrey commented Nov 28, 2015

Move config outside of files (and preferably outside the docroot)

I'm pretty sure this is not possible when you are hosting on Pantheon, since the docroot is also the root of the git repository so you can't put it outside of that, and files is the only area that the web server has permission to update.

So, am I correct in assuming that with Pantheon, you need to keep the config files in the hashed directory in files? Has anyone figured out a way around that?

@quicksketch

This comment has been minimized.

Member

quicksketch commented Dec 1, 2015

So, am I correct in assuming that with Pantheon, you need to keep the config files in the hashed directory in files? Has anyone figured out a way around that?

You could just add an .htaccess file to prevent it from being accessed directly. We're planning on adding one by default in the future over in #706. However I think the hash will stay either way, for sites that are not running Apache for basic security.

@mikemccaffrey

This comment has been minimized.

mikemccaffrey commented Dec 2, 2015

FYI, the approach I ended up using on Pantheon was to leave the active config directory in its default location under the files folder where the pantheon web server can make changes, but to put the staging config directory in the site root under version control.

This actually enforces a workflow that I prefer to the default one where the contents of the staging directory are emptied after import. Since backdrop can't delete the files in the staging directory if they are under version control, I can go to the config management page at any time and see all the active config changes that have been made by users since the last code release, just like how with features you can see what has been overridden from the version in code.

The only messy part is how it generates dozens of "Warning: unlink() permission denied" errors after you hit Import All. We should get CMI to play nice with version controlled config directories, perhaps as part of the 2.x plans in #435.

@quicksketch

This comment has been minimized.

Member

quicksketch commented Dec 2, 2015

The only messy part is how it generates dozens of "Warning: unlink() permission denied" errors after you hit Import All.

We could gracefully handle this one situation. Just suppress errors from unlink with the @ symbol and display a single message instead.

$deleted_all_files = TRUE;
foreach ($config_files as $file) {
  $deleted_all_files = @unlink($file) && $deleted_all_files;
}
if (!$deleted_all_files) {
  backdrop_set_message(t('Failed to deleted some configuration files from the staging directory.'));
}
@heyrocker

This comment has been minimized.

heyrocker commented Dec 2, 2015

Is there an argument to be made that you shouldn't delete after import at all? I know this argument was made during dev on d8 and i was against it, but i might be warming to it now.

@heyrocker

This comment has been minimized.

heyrocker commented Dec 10, 2015

I was trying to remember back to how this whole thing came about, and here's a discussion of the problem.

https://www.drupal.org/node/1697256#comment-6641656

It really stems from "How do you stage deletes?" When a config file previously existed and then disappears, the config system sees this as a delete. There are situations where this can get totally foobar'd if your import directory is not cleaned out after every use. Here is an example:

  1. You import a bunch of stuff, and then your import is left behind.
  2. In your active directory you delete a field.
  3. You copy your config from active to import.
  4. Your deleted field is still there, because you just copied files from one place to another.
  5. When you import your field doesn't get deleted.

Now sure, you can be extremely careful when doing that copy from dir to dir, but we were trying to figure out how to manage that in a more foolproof manner. Note that the management of how to stage deletes was easily the most complicated problem we had to solve in the base system. You can read that entire thread for tons of alternate approaches we tried and finally abandoned.

Just wanted to throw that background out there, for people who might be thinking about trying to work around that directory emptying behavior.

@mikemccaffrey

This comment has been minimized.

mikemccaffrey commented Dec 14, 2015

We could gracefully handle this one situation. Just suppress errors from unlink with the @ symbol and display a single message instead.

Or we could just check if the system has write permission in the staging directory, and present a single clear notice such as "backdrop cannot automatically remove the staging files".

It really stems from "How do you stage deletes?" When a config file previously existed and then disappears, the config system sees this as a delete. There are situations where this can get totally foobar'd if your import directory is not cleaned out after every use.

This is definitely a problem if the user is manually copying config files around, but should not come up if the config is in version control. We may want to think about how to handle each of these two very different use cases in their own fashion.

@mikemccaffrey

This comment has been minimized.

mikemccaffrey commented Dec 21, 2015

@heyrocker: It really stems from "How do you stage deletes?" When a config file previously existed and then disappears, the config system sees this as a delete.

Personally, I'm thinking that a better solution for this would be to export a config file with a list of all the elements that have been deleted, which can be checked on import to see if anything needs to be removed. That seems much more foolproof than assuming that a missing file means that configuration and content should be deleted.

What would the drawbacks be to this approach?

@heyrocker

This comment has been minimized.

heyrocker commented Dec 21, 2015

We actually implemented and later backed out that approach. I highly
recommend you read the linked thread above, it goes into these discussions
in detail, and gives a really good overview of all the pros and cons.

On Mon, Dec 21, 2015 at 1:21 PM, Mike McCaffrey notifications@github.com
wrote:

@heyrocker https://github.com/heyrocker: It really stems from "How do
you stage deletes?" When a config file previously existed and then
disappears, the config system sees this as a delete.

Personally, I'm thinking that a better solution for this would be to
export a config file with a list of all the elements that have been
deleted, which can be checked on import to see if anything needs to be
removed. That seems much more foolproof than assuming that a missing file
means that configuration and content should be deleted.

What would the drawbacks be to this approach?


Reply to this email directly or view it on GitHub
#855 (comment)
.

@sirkitree sirkitree referenced this issue Mar 1, 2016

Merged

switching to puppies and kittens for config #6

2 of 3 tasks complete
@thamas

This comment has been minimized.

thamas commented May 5, 2016

Hi all,

I just started to collect information about and try Backdrop. And I'd like to tell you that it is really strange that such an important feature like this, which is listed in the second position of the Backdrop homepage(!), has not a dedicated documentation section!

So one have to start search for information and arrives to an issue queue where the actuality of the information available is not known…

@docwilmot

This comment has been minimized.

Contributor

docwilmot commented May 5, 2016

Hello, I have personally determined to work only on documentation after version 1.4.0. Sorry about this.

@jenlampton

This comment has been minimized.

Member

jenlampton commented May 5, 2016

@thamas

This comment has been minimized.

thamas commented May 6, 2016

@docwilmot @jenlampton I know that a new release is coming in a few days and I can imagine how hard you work!
My comment is intended to be more general not specified to these very busy days.
Also adding a page to the documentation with one link to this (or a more relvant issue, blogpost etc.) could be a big step forward I think. :)

@serundeputy

This comment has been minimized.

Member

serundeputy commented Oct 23, 2016

I've written a drush command bcex that uses the "The versioned staging directory approach".
That can be found at: backdrop-contrib/drush#74

And I'd be grateful for testers and reviews of that workflow.
This workflow keeps the git status clean and therefore useful.

This downside:

(1) ACCIDENTAL REVERTS ARE POSSIBLE: If there was ever a config change made on production and not actively copied out of the active directory and committed wherever it's supposed to be tracked, the next time code is deployed that production change would be reverted.

Is one I'm used to living with this in the features for deployment workflow and our team just says don't make config changes on production ever!

I'm for documenting this as well. If we can get consensus that using the versioned staging approach is acceptable or even desirable. I would volunteer to write this documentation.

Now with panttheon.yml file allowing for nested docroot this approach will work well with the Pantheon platform as well. I had/have some issues with testing on Pantheon (permissions), but I've got support requests in for them.

thanks,
~Geoff

@xmacinfo

This comment has been minimized.

xmacinfo commented Jan 1, 2017

Is there any update on Full import/export documentation? The import part does not work.

While moving a Backdrop site to a new host, I add to import all the configurations manually by adding all the files in the staging folder. Something more automatic would have been more helpful. :-)

@Graham-72

This comment has been minimized.

Graham-72 commented Jan 2, 2017

@xmacinfo I share your concern that this does not seem to be documented (if it is, where is it?), but the other day I successfully used the import part. It seems to be critical to do things in the right order.

Some time ago a made a personal copy of the procedure to follow - I think it may have been written somewhere by @jenlampton. This is it:

Suggested procedure for moving a site to new server space:

  1. Back-up the existing site by:
    a) Using contributed module ‘Backup and Migrate’ to save site data tables and content as an sql file (only the ‘Quick Backup’ is needed).
    b) Export the current configuration as a gz file by using Configuration Management / Full Export.
    c) Make sure you have an up-to-date copy of the contents of the following directories:
    - files
    - layouts
    - libraries
    - modules
    - themes

  2. Upload into the new server space

  • Backdrop
  • all contributed modules used by the site
  • all files (except the config_.... folder)
  • layouts
  • libraries
  • themes
  1. Install Backdrop as a new site
  2. Activate each module used in the site
  3. Import the configuration gz file exported in step 1.
    The Configuration Management function will first load the json files into the staging folder and then check whether they can be loaded as active files. Any discrepancy, such as a missing module, will halt the process with an error message advising, for example, on which module is missing. Either install the missing module and try again, or if perhaps it is a result of a module that is no longer in use, delete the redundant json file from the staging folder before trying again.
  4. Import the data tables
    Use Backup and Migrate to restore the site’s data tables saved in step 1.
@herbdool

This comment has been minimized.

herbdool commented Mar 22, 2017

I had tried out putting the staging directory into version control, but in the case of a particular site, it can lead to problems. It's probably an edge case, but thought I'd mention it.

This site uses a service to deploy the code. It has the option to deploy via FTP or git. In this case it's deploying via FTP and will only FTP the files which it detects had changed. So what happens is that the staging folder will only contain the files that changed. Backdrop will assume that all the rest of the missing JSON files means the config was deleted. So the sync can't be used to import the new changes; it'll just ruin the site.

So I guess we'll just use a third folder to hold the versioned files and use the export/import tarballs instead, or connect via an ftp app and upload/copy the files from the versioned directory. Though it's bit of a bother in our workflow if we're mostly just using a service to deploy code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment