Software Development Procedures
Draft Software Development Policies and Procedures
SlashRoots Foundation
"To be able to choose between proprietary software packages is to be able to choose your master. Freedom means not having a master. And in the area of computing, freedom means not using proprietary software." - Richard Stallman
Revisions
| Author | Change Description | Date | Approval |
| M. Budram | Original Draft Document Created | Dec. 5, 2014 | |
| M. McNaughton | Revision and Suggestions | Dec. 29, 2014 | |
| M. Budram | Inclusion of Narration | Jan. 7, 2015 | |
The purpose of this section is to create processes that can allow for the establishment of new code or make allowances for changes in an existing code-base. A variety of methodologies exist, but it is the position of SlashRoots that we seek to utilize Agile Processes as it promotes efficient communication, generates maintainable source code, and produces software that is implemented to specification.
So Johnny Appleseed just joined the engineering staff of SlashRoots. He’s fresh out of school and wants to dive right in but at SlashRoots we focus heavily on collaboration and designing for users. This requires adhering to the 5 main design steps:
-
Clear, Concise and Complete Documentation
-
Well maintained and formatted code
-
Ensuring highly resilient and robust code by thoroughly testing
-
Sharing and collaboration using code versioning tools
-
Operating in a **secure **environment
These tools are the common tools across the developers at SlashRoots:
IDE
pyCharm (MacOS X, Windows, Linux) [https://www.jetbrains.com/pycharm/]
wing (Windows, Linux)
[http://wingware.com/]
Database
Postgresql [http://www.postgresql.org/]
Project Management
Asana [https://app.asana.com/]
Team Chat / Virtual Office
Slack [http://slashroots.slack.com/]
Version Control
Git / git-core / gitk
Github [http://github.com]
SourceTree (MacOS X, Windows) [http://www.sourcetreeapp.com/]
Tortoise Git (Windows, Linux) [https://code.google.com/p/tortoisegit/]
As part of the onboarding process of the SlashRoots Labs the engineer will be provided with an @slashroots.org email address that gives you access to all documentation and resources housed and managed by Google.
In order to enhance the collaboration effort of the team we utilize Slack. Slack is a team communication tool that aggregates most software development tools into one centralized application. This reduces the effort to quickly and effectively show project updates, request meetings and find related notes/documents. This tool aggregates information from the slashroots google drive, Asana project planning tool and versioning applications such as Git.
As a SlashRoots engineer it is important that you actively contribute to open-source projects. Therefore you should have already created an online-based profile for Github. This profile will be used to contribute to SlashRoot’s open-source projects and adds to your own personal credentials as an engineer!
SlashRoots utilizes the Google document facility to manage all documents including (but not limited to):
-
Business Development
-
Grants
-
Project Requirements/Information
-
Proposals
This would be a good point for Johnny to get up-to-speed with the operations of the SlashRoots Foundation. The development team uses an Agile approach to software development and as a result any changes made to various prototypes during development phases must be reflected by the requirements docs of the project. At all times the project information must remain current.
Using the Google Drive facility ensures that all revisions and changes are tracked and can be monitored to ensure quality and integrity of the document. This is essential when distributing information to third parties.
The research framework is a living document that defines your research themes and questions, but its focus will evolve and narrow over time, first as feedback is gathered from stakeholders, and then even more so over the course of field research.
While not a record of insights or findings, it is reflective of an increasingly sophisticated understanding of
the context, honing in closer to the actual truth as the research unfolds. Over time, more targeted and more detailed questions arise.
In practice, priorities tend to shift from what was originally put on paper. All practitioners know about
the importance of flexibility and striking a balance between planning and operations. The research framework creates a way to adjust to shifting donor and stakeholder priorities, while also keeping the on-the-ground realities in focus.
Johnny Appleseed has been brought up-to-speed with the requirements for the "Open Potato" project that he has been assigned to. After downloading the tools in the SlashRoots Development Toolkit, he is now assigned to write a module that should allow for the tracking of prices for potatoes. SlashRoots primarily develops open-source projects and is therefore largely collaborative and requires adhering to standards that are widely adopted by the community. Before he starts writing his first line of code he needs to review the steps involved in properly formatting his code.
Proper formatting allows a team to easily maintain a code base, and in python it enforces variable and method scoping. There are also a number of automated document generators that can extract code documentation and display it for easy and quicker readability (example Swagger™ for python). https://helloreverb.com/developers/swagger
This section is based on the recommendations made by the Python team (Warsaw, Coghlan and vanRossum 2001).
Modules should have short, all-lowercase names. Underscores can be used in the module name if it improves readability. Python packages should also have short, all-lowercase names, although the use of underscores is discouraged.
Since module names are mapped to file names, and some file systems are case insensitive and truncate long names, it is important that module names be chosen to be fairly short -- this won't be a problem on UNIX, but it may be a problem when the code is transported to older Mac or Windows versions, or DOS.
When an extension module written in C or C++ has an accompanying Python module that provides a higher level (e.g. more object oriented) interface, the C/C++ module has a leading underscore (e.g. _socket ).
Class names should normally use the CapWords convention.
The naming convention for functions may be used instead in cases where the interface is documented and used primarily as a callable.
Note that there is a separate convention for builtin names: most builtin names are single words (or two words run together), with the CapWords convention used only for exception names and builtin constants.
Function names should be lowercase, with words separated by underscores as necessary to improve readability. mixedCase is allowed only in contexts where that's already the prevailing style (e.g. threading.py), to retain backwards compatibility.
Use the function naming rules: lowercase with words separated by underscores as necessary to improve readability.
Use one leading underscore only for non-public methods and instance variables.
To avoid name clashes with subclasses, use two leading underscores to invoke Python's name mangling rules.
Python mangles these names with the class name: if class Foo has an attribute named __a , it cannot be accessed by Foo.__a . (An insistent user could still gain access by calling Foo._Foo__a .) Generally, double leading underscores should be used only to avoid name conflicts with attributes in classes designed to be subclassed.
Constants are usually defined on a module level and written in all capital letters with underscores separating words. Examples include MAX_OVERFLOW and TOTAL .
Utilize existing framework to operate faster and therefore smarter! Never reinvent the wheel – unless that’s the aim of your project! Some of the framework and packages used by SlashRoots:
DJango - https://www.djangoproject.com/
Password Policy Framwork - https://pypi.python.org/pypi/django-password-policies
Drupal - https://www.drupal.org/
Within a week, Johnny wrote his first module in python! He launches the application but it fails catastrophically. Johnny Appleseed forgot to do any testing! This next section ensures that the products designed and developed are of the high quality that is expected of the SlashRoots Engineering Lab.
This is a type of testing in which discrete units of source code is tested to verify they work as desired. These tests are recommended to run automatically throughout the development process. If the system being developed is written in Python it is recommended that the Python Unit Testing (PyUnit) be used unless another, more supported framework has been chosen by a senior engineer.
In each project a suite of tests should be developed and maintained through the project lifecycle allowing for faster debug times. Each engineering resource is responsible for the development of his/her own unit tests.
Test suites are primarily used to check if requirements developed during the research phases are met by the software development team. When used in conjunction with a continuous integration tool such as Travis it allows for more robust deployments that are of a high quality. Travis runs a series of tests using the test suites developed against a target environment to ensure that the requirements are adhered to.
*An example of this is an API where a session should expire after 1 hr. To test this Travis may attempt to set up a session then expire the session by using the database. If the session is still active after manually expiring the session then the test will fail. *
Working on projects with multiple collaborators and large volumes of files can become very difficult – especially because people can change the same files. Code versioning tools allow us to manage changes to files giving the ability to compare, restore and merge revisions done on files.
We understand that you’re a new and learning how to take advantage of code versioning systems such as Git and CVS is difficult and has a fairly steep learning curve. We recommend that before you begin you should research some visual tools such as sourceTree for MacOSX or tortoiseGit for Windows – this makes life easier and you can see exactly what is happening before you take on the scary terminal!
These files are NOT allowed in repositories and should be managed within the .gitignore file at the root of the repository. Changes to binary files give versioning systems a very hard time to manage and usually results in large differences between commits and therefore a larger footprint for each commit. Allowances should be made for static binary files necessary in the code project such as images or videos. In a python the following files should be included in the .gitignore file (https://github.com/CodeForTheCaribbean/HarvestAPI/blob/master/.gitignore).
Each developer is responsible for managing his/her library files (for python based projects). The lib/ and lib64/ folders are by default excluded from the repository. Also with the pip package manager, this is usually an automated process as long as a requirements file is present in the repository and actively maintained.
This is essential for the documentation of versions throughout the project’s history (GIT 2014). Like most VCSs, Git has the ability to tag specific points in history as being important. Typically people use this functionality to mark release points (v1.0, and so on). The creation of an annotated tag can help with the generation of change release notes. These notes document the changes to the project between versions.
Creating an annotated tag in Git is simple. The easiest way is to specify -a when you run the tag command:
The -m specifies a tagging message, which is stored with the tag. If you don’t specify a message for an annotated tag, Git launches your editor so you can type it in (recommended).
At the core, the development model is greatly inspired by existing models out there. The central repo holds two main branches with an infinite lifetime:
-
master
-
develop
The master branch at origin should be familiar to every Git user. Parallel to the master branch, another branch exists called develop.
We consider origin/master to be the main branch where the source code of HEAD always reflects a production-ready state. When the source code in develop is deemed stable it should be merged back into master, tagged with the release number and a nice detailed message describing the changes between the previous production release and the current one. This is VERY important, so that theoretically, we could use a Git hook script to automatically build and roll-out our software to our production servers every time there was a commit on master.
**Figure 1 - Basic Git Model **
Merging is Git's way of putting a forked history back together again. The git merge command lets you take the independent lines of development created by git branch and integrate them into a single branch.
Firstly before proceeding we must define the various branches and their definitions in the context of software development:
-
master – This is seen as the production ready branch. Any changes done to this branch will include new features and fixes to correct security vulnerabilities and bugs.
-
develop – This is the main branch where new features are integrated and tested. All tests must be performed here before merging into production ready branch (master).
-
feature – This is a branch off develop and is usually treated as a supporting branch and is used to develop new features for the upcoming or future releases. In the case of experiments this is important as failures can be easily discarded without affecting develop.
-
hotfix – This is another supporting branch of master and is used in emergency cases where critical and severe bugs are quickly corrected and put back into production. The hotfix MUST also be incorporated into develop to prevent undoing changes to master at the time of committing a new feature.
As you can tell, there are several reasons to merge branches. Some of the main reasons:
-
The engineer has completed a new feature and wants to incorporate it back into the code base. He/she would merge the feature branch back into develop, for future compliance checks and eventual incorporation into the production release.
-
A severe/critical defect has been corrected in a hotfix branch and it is necessary to move it back into production and develop.
-
An experiment went well and has been approved to be a part of the project by a senior engineer.
The technical details about merging are further described by Atlassian at https://www.atlassian.com/git/tutorials/using-branches.
Figure 2 - More complex Git model (usually as the project evolves and team increases in size) (Driessen 2010)
Since the master branch is denoted as the production-ready branch, it is necessary to have a high level of control over the merge request to this branch. The acceptance of the merge request is pending an acceptance testing of the code-base by a senior engineer, done on an infrastructure similar to that of the production environment.
This step again is necessary especially if there are hooks to the master branch of the repository allowing for automated deployment into production. A mistake here can be devastating to mission critical applications and especially damaging if there are established service-level agreements (SLA).
In the acceptance testing the software is checked against requirements and specifications to determine if the software successfully conforms. If the software is rejected, the senior overseeing engineer or project owner must make a list of reasons and suggestions to the developer(s) involved. A request may be rejected if it conflicts with:
-
Project road map
-
Business requirements
-
Bugs or issues with code-base
-
Poorly documented code
-
Security Vulnerabilities
-
UX/UI issues
-
Deprecation Violations
-
Architectural Issues
-
Copyright Infringement(s)
-
Proprietary Software Package Usage
-
IP Violoations
A change management process is designed to control and manage the propagation of software and infrastructure changes to a production environment. Different kinds of changes may warrant different refinements or variants of the process, depending on the nature of the changes and their potential impacts. This process prevents the uncoordinated changes that might not be in-line with the requirements outlined by the owners of the system. It also provides feedback into our project management allowing for proper planning for any changes to the system being developed.
The following practices are integral to a SlashRoot’s change management process:
-
Authorization—Any changes to the production environment should be subject to review by a responsible party within the organization. The depth of the review performed should align with the magnitude of the proposed change.
-
Auditability—The establishment of procedures for communicating requirements, changes and functionality related to their proprietary software and technical infrastructure. The team should also maintain a historical audit trail of material changes made to their proprietary software, allowing them to accurately determine:
-
When a change was made
-
Who made the change
-
The nature of the change
-
Document Management tools have added features that allow for clear and easy document auditing. One such recommended tool is Google™ Drive.
This section of the document is primarily focused on the development of a secure model allowing for the intelligent access of resources developed and managed by the SlashRoot’s Foundation and/or its subsidiaries. This ensures that data will be:
-
Protected from an outside adversary
-
Dangerous actions are limited to those internal to the project
-
Safely disseminated by approved vendors/users
-
Accessible in an auditable fashion
-
Protected by an disasters either natural or unnatural
-
Remain with a high level of integrity and accuracy
It is recommended that users or applications utilizing the web-services be authenticated using standards such as OAuth spec 1.0 or 2.0 (Brail 2010). This allows a more controlled method of giving and revoking access to rogue users or applications. In order to facilitate OAuth registration it is necessary to provide a separate authentication for users – thus providing a separation of users to the resources and applications utilizing the services.
Detailed logs of user usage must be maintained throughout the project lifetime, ensuring that, information about what the user(s) have done can be presented to the relevant authorities in accordance with the laws of the jurisdiction(s) in which SlashRoots operates. These logs can also be used to identify user(s) with malicious intentions and revoke any accesses granted by SlashRoots.
Below are some guidelines that all engineers must follow:
-
Configuration files must NOT be published to any open repositories as outsiders with sufficient knowledge can compromise infrastructure and services supported by SlashRoots.
-
Any action to remove data in an existing dataset must have the approval of a senior engineer
-
Storage of access codes must be done utilizing industry standards ensuring information privacy
-
At no point must a password be stored or transmitted in the clear
-
Utilization of data sanitation libraries must be done in order to prevent database breaches
-
The credentials used for database authentication must not reside in the main, executing body of the program's source code in clear text
-
Database credentials must not be stored in a location that can be accessed through a web server
All passwords should meet or exceed the following guidelines:
-
Contain at least 12 alphanumeric characters.
-
Contain both upper and lower case letters.
-
Contain at least one number (for example, 0-9).
-
Contain at least one special character (for example,!$%^&*()_+|~-=`{}[]:";'<>?,/).
If an in-house repository is being used, each developer must access this resource through a secure socket layer utilizing his/her pre-shared public key. No other authentication is acceptable for repository-level action. A public key housed on the SlashRoots infrastructure can be removed if the machine is no longer a contributing member of the foundation and/or subsidiaries.
Under no circumstances is a private key to be shared with any external or internal entity. This can compromise the information moved between the owner of the key and any communicating entity – leading to a breakdown of security.
Passphrases generally are used for public/private key authentication. A public/private key system defines a mathematical relationship between the public key that is known by all, and the private key, that is known only to the user. Without the passphrase to unlock the private key, the user cannot gain access.
A passphrase is similar to a password in use; however, it is relatively long and constructed of multiple words, which provides greater security against dictionary attacks. Strong passphrases should follow the general password construction guidelines (SANS Institute 2014) to include upper and lowercase letters, numbers, and special characters (for example, TheTrafficOnThe101Was*&!$ThisMorning!).
This section hopes to amalgamate all the various processes outlined in the document, providing a more streamlined workflow. All work done at the SlashRoots Labs is based on a Collaborative effort ensuring success and high quality products and services.
"Alone we can do so little; together we can do so much" - Helen Keller
Based on the project’s requirements a senior engineer/architect will communicate the system’s design to the team. This design gets broken down into smaller components that get assigned to the engineering team.
The tasks details for each developer is housed in Asana in order to manage the team in collaborating on projects. As tasks are completed the engineer is required to enter the status of the task and/or any issues that he/she has encountered. This is important as Asana collects the throughput of all developers involved in the project and will be considered a part of the engineer’s performance on the project(s).
During the development phase of the project, engineers working on the project are required to ensure their functionality developed is inline with the requirement developed for the component. The development of a suite of test scripts is necessary during the evolution of the project. The test suite will be used to aid in the continuous integration efforts at the SlashRoots Labs. If tests are failed in the project this could mean that the service/product being delivered does not meet the functionality described by the requirements document or by user-based research.
There are times that there are changes that are necessary to the project. These changes are managed by an elected change review board (typically the project leads). These changes are usually major changes to the described functionality in the requirements. Accepted changes to functionality must be documented clearly and the necessary modifications done to design/specification documents.
Continuous Integration tools such as Travis ensure that commits being pushed to the repository is of high quality ensuring that the checks against functionality of the project are done. This prevents bad-programmatic efforts from being staged into the project repository. This also prevents bad/malicious code from making it’s way into the production environments.
Communication is important during research, designing, dissemination and development of the project. Tools such as Slack can minimize the times spent drafting emails giving project updates or sending documents while increasing the efficiency of the organization - by making communication and information accessibility easier. Slack has dedicated channels for the organization allow for information to be categorized and relevant to the consumer. It allows a user to subscribe to these channels allowing the user to be a part of a discussion where information is shared.
Reminder: We are a firm believer in the Agile Development approach to development of software, but without communication the approach can lead to poorly designed systems. Documentation is an important part of the communication process. As decisions are made during the development/design process it must be incorporated into the relevant documents for the systems and/or stakeholder updates.
Throughout the lifecycle of the project, ensure that the project tasks are completed and updated in order to accurately track the progress of the project/developer.
As a final test of the system we seek to put the system through its paces by running a thorough performance test to establish a set of measurable statistics that can be used by the SlashRoots Legal team in the formulation of a Service-Level Agreeement (SLA). This test must be made on an infrastructure that matches that of the production infrastructure.
Brail, Gregory. Why you need OAuth for your API or App. April 8, 2010. https://blog.apigee.com/detail/why_you_need_oauth_for_your_api_or_app (accessed December 8, 2014).
Driessen, Vincent. A successful Git branching model. January 25, 2010. http://nvie.com/posts/a-successful-git-branching-model/ (accessed December 8, 2014).
GIT. Git Basic Tagging. December 8, 2014. http://git-scm.com/book/en/v2/Git-Basics-Tagging (accessed December 8, 2014).
SANS Institute. "Information Security Policies." SANS. June 1, 2014. http://www.sans.org/security-resources/policies/general#password-construction-guidelines (accessed December 8, 2014).
Warsaw, Barry, Nick Coghlan, and Guido vanRossum. Naming Conventions Python. Python.org. July 5, 2001. https://www.python.org/dev/peps/pep-0008#naming-conventions (accessed December 5, 2014).
