Skip to content

About Pullpush

bailey86 edited this page Jun 16, 2013 · 36 revisions
Home About PullPush Community Setup Guide User Manual

Table of Contents

IMPORTANT

The scripts are designed to work with four project types; drupal7, drupal6, symfonymysql and phpmysql. They are also designed to work with two server types; debian and sles - and two Apache types; ITK and non-ITK.

Please note, currently only drupal7 on debian with Apache type ITK is fully supported in all the scripts. For Drupal please note that non-ITK Apache can be used by simply changing the permissions on the sites/default/files directory after running certain scripts so that the web server can read/write to them.

It will not take much work to complete the scripts for all four project types and for both server types and for both Apache types - but this has not yet been completed.

It is planned that these scripts can be expanded to handle other project types (codeignighter, zend etc) and other server types.

Currently, please look closely at the scripts to see how they suit your needs.

Introduction

This page describes the Pullpush framework.

Thanks are due to SSVC for sponsoring this project and to Karim Kronfli for commissioning the project.

Pullpush is a framework for developing web applications. It is designed to be used on a development server to enable a team of web developers to work together on websites and web applications.

Although Pullpush has originally been developed to be used for Drupal based sites it can be extended to work with websites and web applications based on other types of websites and web applications.

Quickstart

Pullpush is basically a set of scripts which are put on to a (usually LAMP type) development server to help with development and deployment of web applications. Developers each have an account on the server and carry out development work directly on the server. There are also some specific user accounts set up to hold various versions of sites such as beta, livecopy etc.

Once a new version of a site is ready to be deployed the scripts are used to merge live data with this latest version. This new merged site (latest beta site version merged with live data) is then made available to be the new live site.

An important point to note is that the initial site could be pulled down as a vanilla copy of a framework or it could be created from an existing website/web application.

This diagram gives an overview of the framework and processes. More details are available in the Setup Guide and the User Manual.

NB - The names in bold next to the arrows are the script names.
NB - A key script is pushlivecopydataintodeploy - which means 'push live copy data into deploy'. For Drupal sites this step installs and then uses the Drupal module Data Export Import. This module enables the three main Drupal data types of users, taxonomy terms and nodes to be exported from one site and imported into another one.

Web applications are everywhere

Web applications are now widely used across the internet to provide access to data and to enable data to be updated. They also enable other systems to be able to update and consume data via the use of web technologies such as REST, SOAP, XML etc.

Many modern websites allow website visitors to add data in the form of comments, messages, etc. They also use content management systems to enable non-technical website editors to be able to add and update content such as pages, articles etc. Most of the time these content management systems are web based and in many cases are closely integrated with the framework of the website which displays the information.

In fact, all websites can be thought of as web applications. On very simple websites the users request pages and the website then serves up those pages to the user's web browser.

Teamwork needed

Many large websites and web applications need to be maintained by teams of developers and designers. This means that there needs to be a development/deployment structure in place to enable the team members to be able to work together to maintain and update the primary live websites.

Modern web application frameworks such as Drupal, Symfony, Codeigniter, etc use patterns such as Presentation–abstraction–control (PAC) and Model–view–controller (MVC) to cleanly separate certain functions of a web application. However, for large teams these patterns do not provide enough separation on their own to enable developers to work efficiently together.

A development/deployment structure is needed to enable developers to work efficiently and enable them to be able to collaborate easily and safely.

Add live data to the mix

Many websites contain a large amount of data which is supplied by website visitors or website editors and this data is being continuously updated. For example, a news organisation could have reporters located worldwide who are adding articles to its website at all times of day and night.

Integrating large amounts of existing live data with substantial website changes needs to be managed properly to avoid data loss and site downtime.

Using Pullpush for development and deployment

Pullpush is deisgned to be a framework which can be used for web application development and deployment.

Development

Developers may be asked to provide changes to a site's functionality or design. These could range from relatively minor tweaks to complete overhauls. Also, security vulnerabilities and bug fixes are regularly released for the tools and frameworks used to build websites. These need to be applied regularly to help prevent sites from being compromised.

The Pullpush system enables developers to carry out their work effectively by enabling them to quickly and easily carry out the following tasks:

  • Make a copy of a live site for testing minor updates and bug fixes.
  • Have new sites created based on the common frameworks used by the team.
  • Create a beta site from a current site to enable other developers to work on it as a team.
  • Make a copy of the current beta version as their own development copy.
  • Refresh their development version from the beta version if needed.
  • Merge their updates and enhancements into the beta version.
  • Merge current live data into the current beta for testing and review.
These tasks are carried out by running single bash scripts.

Deployment

The approach used by Pullpush is to leave the existing live site untouched during the deployment process.

A beta copy of the site will have been worked on during the development phase. Pullpush implements procedures which extract the live content data from the live site and then merge this data into a copy of the beta site. Once the data export/import has been completed the web server configuration is changed to make it use this new merged site as the new live site.

This approach has the advantage that large changes to the design and functionality can be implemented and tested on the beta site. This beta site can be tested with the full amount of live data imported. Another advantage is that if there is some unforeseen problem with the new merged version of the website then the existing live website can be reactivated.

Key aspects of web applications to be considered

There are various key aspects of websites and web applications which need to be considered when planning and implementing development and deployment procedures.

General web application structure

The underlying structure of most web applications is relatively straightforward. It usually consists of a collection of website files (PHP files, templates, javascript files, CSS files etc) and an underlying database. By making an exact copy of the files and the underlying database it is relatively easy to create an exact copy of most websites and web applications.

Server setup

The underlying server setup platform needs to be considered when setting up development and deployment procedures. This especially needs to be considered when deploying a site to a server which has different versions of the underlying services (Apache, MySql etc).

Also, when a site is copied to another server there may be some issues with a website's specific setup such as hard-coded paths etc. For development and deployment to work well there should be a minimal number of server specific settings.

Licensing - help or hindrance

Since most web applications run on software which is licensed under open source licenses it is possible to make as many copies of the websites as required without any licensing issues. This removes a whole area of concern when replicating websites as developers do not need to work within any licensing restrictions.

Databases - can they be moved around?

The databases behind web applications can usually be dumped to files and then reloaded into other databases which are used behind the same website or behind another website. With most web applications the ability to copy and move databases is relatively easy and no changes are normally needed to the database itself.

Data files - non database data

Very often sites store data files as part of the site content. These are usually image files which are used in the site content pages or can be files which are made available for download such as PDF files.

It is important that these data files are taken into consideration when moving content data between sites.

Site settings - special files

Very often web applications have site specific settings stored in a file. This can sometimes just be the database connection settings but can also be other site specific settings. When carrying out development and deployment it is important that these settings files are handled carefully.

Development and deployment principles

Here we will make clear some of the key principles which are important to the Pullpush framework.

User groups

In general we can think in terms of there being three main groups of people who access websites and web applications.

Site visitors

These users view websites as a source of information. They may add to the content of a site by posting comments or adding other simple content such as messages etc.

Site editors

These users may not be technical but are responsible for updating website content. Editors are generally responsible for more advanced content such as adding new articles or updating pages.

Site developers

These people are responsible for building, maintaining and enhancing the websites.

Site continuity

Websites are normally expected to be available at all times so any website downtime needs to be kept to an absolute minimum. This may be less stringent for access by site editors to update content.

Fallback plans should be in place in case there are problems with deployment to be able to get back to a known good working website.

Development copies

It should be quick and easy for developers to be able to make copies of the website to carry out development work on. This would normally be copied from the current beta version by running a script. This makes it much easier for developers to try out different options as they know it is easy to re-run the script and get a fresh development copy from the beta.

Under Pullpush the development copies are held on the development server and so can be made available to project managers and other stakeholders. This means that developers can work closely with other members of their company/team as development of a particular function/feature/fix cna be tested and reviewed before it is pushed up to the beta copy.

Beta sites - the developer's shared master copy

Once a website is live any further development of that site should be carried out on developers own development copies of the website.

Once a developer is satisfied with the changes which have been made then these changes should be pushed up to a shared beta copy of the website. This beta version should be treated as the 'Gold Standard' and only fully developed and tested changes should be pushed up to the beta version.

The beta version is used as the master copy when new developers join the team. To enable the new developers to start working they will be provided with a development copy which is a copy of the current beta version.

This beta version can be made available for general testing. Site editors would be able to access the beta version and test/review changes which have been made.

Testing the new site version

It should be possible to make a copy of the beta site and merge in a recent dataset of live site data. This will enable stakeholders to be able to carry out testing with realistic amounts of site data.

Deployment to live - it should be easy

So the beta site has been developed extensively and now has features and functions which need to be deployed to the live site.

It should be easy for developers and systems adminitrators to be able to create a site which contains both the latest developments which have been pushed into the beta site and the current data from the live site. These processes should be scripted so they are easy to run and they should be tried and tested.

If the update/upgrade process is made relatively easy then updates and upgrades will be carried out more frequently which is better for site stability than infrequent, large changes.

Documentation of updates and changes

All changes and development of a website should be properly documented. The development and deployment strategy should make it easy for developers to document their changes.

If the development changes are regular and atomic then it should be possible to maintain a comprehensive change log. Revision numbers should be used to relate the documentation to the code changes. Documentation is especially important when the changes are mainly implemented using the admin interface and involves minimal code changes.

Documentation should also be linked to the logs used by any code versioning tools used.

Avoiding change conflicts

Strategies will have to work in a way which avoids code conflicts between changes made by different developers. Even though developers may be working on very different aspects of a website it is still possible for the changes to conflict and to cause serious problems.

Frequent updates and changes

The strategies used should encourage changes to be atomic and frequent. If is is easy for a developer to pull down a copy of the beta site and to then push development changes up to the beta site then they are more likely to make smaller changes more often. This would make documentation easier as the beta site change log would show each new feature/bugfix etc as a new revision.

Also, if it easy to update the live site regularly then there is less likely to be changes which cause problems. Also, features/enhancements requested would be more quickly incorporated to the live version of the site.

Tools

The Pullpush platform uses some other tools to carry out various functions.

Subversion

The code versioning tool subversion is used to hold a central code repository and this is used to synchronise the website files between development and beta copies.

Subversion can also provide a basis for versioning and the revision numbers can be used as the basis for documenting the development of websites.

Git

The code versioning tool git can also be used to version and synchronise files.

Drush

The Drush tool is extremely useful when developing and deploying Drupal based websites and web applications.

Drush is a command line tool developed specifically for Drupal which can be used to carry out various tasks on Drupal websites.

There are a large number of tasks which can be carried out by Drush on the command line. The tasks which are useful for development and deployment include being able to sychronise databases and files between websites and being able to clear out caches. Drush can also be used to install/enable/disable modules and to switch websites in and out of maintenance mode.

Data Export Import module

The Data Export Import module enables a Drupal site live data to be exported to files and for these files to be imported into Drupal sites. The data exported includes Users, Taxonomy Terms and Nodes. Normally, this module is used in development and deployment strategies to extract data from a live site and to then import this data into updated versions of the site.

It can also be used to dump out a known good set of data before development or testing is carried out. This data can then be re-imported to revert the data back to its original state.

You can’t perform that action at this time.