|Home||About PullPush||Community||Setup Guide||User Manual|
Table of Contents
The scripts are designed to work with four project types; drupal7, drupal6, symfonymysql and phpmysql. They are also designed to work with two server types; debian and sles - and two Apache types; ITK and non-ITK.
Please note, currently only drupal7 on debian with Apache type ITK is fully supported in all the scripts. For Drupal please note that non-ITK Apache can be used by simply changing the permissions on the sites/default/files directory after running certain scripts so that the web server can read/write to them.
It will not take much work to complete the scripts for all four project types and for both server types and for both Apache types - but this has not yet been completed.
It is planned that these scripts can be expanded to handle other project types (codeignighter, zend etc) and other server types.
Currently, please look closely at the scripts to see how they suit your needs.
This page describes the Pullpush framework.
Pullpush is a framework for developing web applications. It is designed to be used on a development server to enable a team of web developers to work together on websites and web applications.
Although Pullpush has originally been developed to be used for Drupal based sites it can be extended to work with websites and web applications based on other types of websites and web applications.
Pullpush is basically a set of scripts which are put on to a (usually LAMP type) development server to help with development and deployment of web applications. Developers each have an account on the server and carry out development work directly on the server. There are also some specific user accounts set up to hold various versions of sites such as beta, livecopy etc.
Once a new version of a site is ready to be deployed the scripts are used to merge live data with this latest version. This new merged site (latest beta site version merged with live data) is then made available to be the new live site.
An important point to note is that the initial site could be pulled down as a vanilla copy of a framework or it could be created from an existing website/web application.
- NB - The names in bold next to the arrows are the script names.
- NB - A key script is pushlivecopydataintodeploy - which means 'push live copy data into deploy'. For Drupal sites this step installs and then uses the Drupal module Data Export Import. This module enables the three main Drupal data types of users, taxonomy terms and nodes to be exported from one site and imported into another one.
Web applications are now widely used across the internet to provide access to data and to enable data to be updated. They also enable other systems to be able to update and consume data via the use of web technologies such as REST, SOAP, XML etc.
Many modern websites allow website visitors to add data in the form of comments, messages, etc. They also use content management systems to enable non-technical website editors to be able to add and update content such as pages, articles etc. Most of the time these content management systems are web based and in many cases are closely integrated with the framework of the website which displays the information.
In fact, all websites can be thought of as web applications. On very simple websites the users request pages and the website then serves up those pages to the user's web browser.
Many large websites and web applications need to be maintained by teams of developers and designers. This means that there needs to be a development/deployment structure in place to enable the team members to be able to work together to maintain and update the primary live websites.
Modern web application frameworks such as Drupal, Symfony, Codeigniter, etc use patterns such as Presentation–abstraction–control (PAC) and Model–view–controller (MVC) to cleanly separate certain functions of a web application. However, for large teams these patterns do not provide enough separation on their own to enable developers to work efficiently together.
A development/deployment structure is needed to enable developers to work efficiently and enable them to be able to collaborate easily and safely.
Many websites contain a large amount of data which is supplied by website visitors or website editors and this data is being continuously updated. For example, a news organisation could have reporters located worldwide who are adding articles to its website at all times of day and night.
Integrating large amounts of existing live data with substantial website changes needs to be managed properly to avoid data loss and site downtime.
Pullpush is deisgned to be a framework which can be used for web application development and deployment.
Developers may be asked to provide changes to a site's functionality or design. These could range from relatively minor tweaks to complete overhauls. Also, security vulnerabilities and bug fixes are regularly released for the tools and frameworks used to build websites. These need to be applied regularly to help prevent sites from being compromised.
The Pullpush system enables developers to carry out their work effectively by enabling them to quickly and easily carry out the following tasks:
- Make a copy of a live site for testing minor updates and bug fixes.
- Have new sites created based on the common frameworks used by the team.
- Create a beta site from a current site to enable other developers to work on it as a team.
- Make a copy of the current beta version as their own development copy.
- Refresh their development version from the beta version if needed.
- Merge their updates and enhancements into the beta version.
- Merge current live data into the current beta for testing and review.
The approach used by Pullpush is to leave the existing live site untouched during the deployment process.
A beta copy of the site will have been worked on during the development phase. Pullpush implements procedures which extract the live content data from the live site and then merge this data into a copy of the beta site. Once the data export/import has been completed the web server configuration is changed to make it use this new merged site as the new live site.
This approach has the advantage that large changes to the design and functionality can be implemented and tested on the beta site. This beta site can be tested with the full amount of live data imported. Another advantage is that if there is some unforeseen problem with the new merged version of the website then the existing live website can be reactivated.
There are various key aspects of websites and web applications which need to be considered when planning and implementing development and deployment procedures.
The underlying server setup platform needs to be considered when setting up development and deployment procedures. This especially needs to be considered when deploying a site to a server which has different versions of the underlying services (Apache, MySql etc).
Also, when a site is copied to another server there may be some issues with a website's specific setup such as hard-coded paths etc. For development and deployment to work well there should be a minimal number of server specific settings.
Since most web applications run on software which is licensed under open source licenses it is possible to make as many copies of the websites as required without any licensing issues. This removes a whole area of concern when replicating websites as developers do not need to work within any licensing restrictions.
The databases behind web applications can usually be dumped to files and then reloaded into other databases which are used behind the same website or behind another website. With most web applications the ability to copy and move databases is relatively easy and no changes are normally needed to the database itself.
Very often sites store data files as part of the site content. These are usually image files which are used in the site content pages or can be files which are made available for download such as PDF files.
It is important that these data files are taken into consideration when moving content data between sites.
Very often web applications have site specific settings stored in a file. This can sometimes just be the database connection settings but can also be other site specific settings. When carrying out development and deployment it is important that these settings files are handled carefully.
Here we will make clear some of the key principles which are important to the Pullpush framework.
In general we can think in terms of there being three main groups of people who access websites and web applications.
These users view websites as a source of information. They may add to the content of a site by posting comments or adding other simple content such as messages etc.
These users may not be technical but are responsible for updating website content. Editors are generally responsible for more advanced content such as adding new articles or updating pages.
These people are responsible for building, maintaining and enhancing the websites.
Websites are normally expected to be available at all times so any website downtime needs to be kept to an absolute minimum. This may be less stringent for access by site editors to update content.
Fallback plans should be in place in case there are problems with deployment to be able to get back to a known good working website.
It should be quick and easy for developers to be able to make copies of the website to carry out development work on. This would normally be copied from the current beta version by running a script. This makes it much easier for developers to try out different options as they know it is easy to re-run the script and get a fresh development copy from the beta.
Under Pullpush the development copies are held on the development server and so can be made available to project managers and other stakeholders. This means that developers can work closely with other members of their company/team as development of a particular function/feature/fix cna be tested and reviewed before it is pushed up to the beta copy.
Once a website is live any further development of that site should be carried out on developers own development copies of the website.
Once a developer is satisfied with the changes which have been made then these changes should be pushed up to a shared beta copy of the website. This beta version should be treated as the 'Gold Standard' and only fully developed and tested changes should be pushed up to the beta version.
The beta version is used as the master copy when new developers join the team. To enable the new developers to start working they will be provided with a development copy which is a copy of the current beta version.
This beta version can be made available for general testing. Site editors would be able to access the beta version and test/review changes which have been made.
It should be possible to make a copy of the beta site and merge in a recent dataset of live site data. This will enable stakeholders to be able to carry out testing with realistic amounts of site data.
So the beta site has been developed extensively and now has features and functions which need to be deployed to the live site.
It should be easy for developers and systems adminitrators to be able to create a site which contains both the latest developments which have been pushed into the beta site and the current data from the live site. These processes should be scripted so they are easy to run and they should be tried and tested.
If the update/upgrade process is made relatively easy then updates and upgrades will be carried out more frequently which is better for site stability than infrequent, large changes.
All changes and development of a website should be properly documented. The development and deployment strategy should make it easy for developers to document their changes.
If the development changes are regular and atomic then it should be possible to maintain a comprehensive change log. Revision numbers should be used to relate the documentation to the code changes. Documentation is especially important when the changes are mainly implemented using the admin interface and involves minimal code changes.
Documentation should also be linked to the logs used by any code versioning tools used.
Strategies will have to work in a way which avoids code conflicts between changes made by different developers. Even though developers may be working on very different aspects of a website it is still possible for the changes to conflict and to cause serious problems.
The strategies used should encourage changes to be atomic and frequent. If is is easy for a developer to pull down a copy of the beta site and to then push development changes up to the beta site then they are more likely to make smaller changes more often. This would make documentation easier as the beta site change log would show each new feature/bugfix etc as a new revision.
Also, if it easy to update the live site regularly then there is less likely to be changes which cause problems. Also, features/enhancements requested would be more quickly incorporated to the live version of the site.
The Pullpush platform uses some other tools to carry out various functions.
The code versioning tool subversion is used to hold a central code repository and this is used to synchronise the website files between development and beta copies.
Subversion can also provide a basis for versioning and the revision numbers can be used as the basis for documenting the development of websites.
The code versioning tool git can also be used to version and synchronise files.
The Drush tool is extremely useful when developing and deploying Drupal based websites and web applications.
Drush is a command line tool developed specifically for Drupal which can be used to carry out various tasks on Drupal websites.
There are a large number of tasks which can be carried out by Drush on the command line. The tasks which are useful for development and deployment include being able to sychronise databases and files between websites and being able to clear out caches. Drush can also be used to install/enable/disable modules and to switch websites in and out of maintenance mode.
The Data Export Import module enables a Drupal site live data to be exported to files and for these files to be imported into Drupal sites. The data exported includes Users, Taxonomy Terms and Nodes. Normally, this module is used in development and deployment strategies to extract data from a live site and to then import this data into updated versions of the site.
It can also be used to dump out a known good set of data before development or testing is carried out. This data can then be re-imported to revert the data back to its original state.