Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
OPEN QA at Eucalyptus
What is the main objective?
Open QA for Eucalyptus is to coordinate the Quality Assurance effort of Eucalyptus development with the community, welcoming active participation in improving Eucalyptus system.
What will be open?
Open QA for Eucalyptus includes the following areas:
Share all the build and test scripts with the community via Github
- Currently eutester is in the open
- Make all QA-related source code available on Github, which include:
- euca_builder - QA-Server's build scripts for various Linux distribution.
- auto_pilot - Eucalyptus QA-System's testunit framework and sequencer.
- testunits - All available testunits written by various Eucalyptus developers.
- other QA tools - Other scripts that are part of Eucalyptus QA-System and QA tools
Provide a collaborative space to actively engage with the community to improve the quality of Eucalyptus.
- Use various existing communication platforms to welcome contributions
- Jira: Bug filing and tracking.
- engage.eucalyptus.com: Forums and blogs.
- IRC: Discussions and Q&As.
- Accept contributions for test cases and test plan
- Allow community members to specify which test cases or test plans they would like to include in the QA process
- Testlink: Sharing the test cases and test plans
Achieve complete transparency in QA process for Eucalyptus
- Allow community members to engage in day-to-day QA activities for Eucalyptus
- Display all the QA Test Results Pages via Open QA Website on qa.eucalyptus.com in a daily basis.
- Twice a day all the test results, including the logs, will be pushed out in the open.
- Welcome contributions, criticisms, and feedback based on Eucalyptus QA activities.
- See Open Contribution above
Preview of Open QA
Open QA Page
Open QA Prototype Page:
- The link About will provide detailed information on the contents in Open QA Page.
- The link Contact will invite visitors to Eucalyptus IRC channels.
- The button View on each test-box will bring up the corresponding QA Test Result Page, which is also what Eucalyptus developers use to determine the status of the code.
- Each test will be marked with the tags that display special Eucalyptus system environments, such as SAN, VMWARE, PKG, QA, HA, etc.
- The pictures of the developers will be displayed in the test-boxes to add human element to the activities on the page, encouraging community members to engage with Eucalyptus developers directly.
- Due to the storage constraint, only 7-day worth of the test results, include the log files, will be stored and displayed for now.
QA Test Result Page
A Sample of QA Test Results Page:
- The table above describes the environment and configuration of the test.
- The Result Matrix summarizes the results of the test sequence by displaying passed or failed
- The link to the result, passed or failed, points to the directories that hold the log files and artifacts as well as the source-code for the testunit.
- The link to the name of each testunit opens up a page that provides more descriptions on the exact operations of the testunit.
- Through these pages, visitors will have access to GIT repositories that provide source-code for all testunits.
What are the impacts?
- Crowd-sourcing of QA
- Build trust in quality of Eucalyptus among the community members
- Stay committed to the openness in Eucalyptus development
- Expose various levels of immaturity in the current QA process
- Mis-intepertation of QA results by competitors and customers
What operations will be involved?
The daily operations would involve pushing the test results and their log files to qa.eucalyptus-systems.com once or twice a day.
1. How many tests are running per day?
In a given day, there are usually 8 to 10 tests running -- launched by QA engineers and developers.
2. How much log is generated per test?
Depending on the test configuration and its failure counts, a test produces between 100M to 1G of log files.
What resources will be involved?
Running on AWS as instances.
Backend - Log File Server
on Eucalyptus Production Cloud at Core sites.
-> Embracing Hybrid Cloud between AWS and Euca
Recipe script to allow easy restart of the instance automagically
Instance space on production cloud
- Which size? Small? medium?
Bucket or volume space to hold results
- From the information above, there could be up to 10GB or logs generated per day.
- Suspect that 10GB a day is a lot more than what we can handle at CoreSite
- Need to either move the logs somewhere else, or just post results without logs, or have logs shown 'on request' only.
- Should we use AWS S3?
How much bandwidth is needed between HQ <-> CoreSite for this?
- If we want the logs we are talking quite a few GBs a day, without logs, how much bw are we talking?
Ownership of the instance
- Who is responsible to keep it up and running?
- QA-team, IT-team or Community-team
What is it going to look like in 6 months?
In Need Now
1. More detailed information that explains:
- What is each sequence doing?
- What is each testunit doing?
- How does the QA system work?
2. Make all build and test scripts available to the community on Github.
3. Set up online mechanisms that allow the community members to engage directly with the QA process - i.e. comments, posts, contributions, etc.
Submitted by Vic
Q: What is the page written in?
Q: Is it open as well?
A: Yes, it will be open-sourced; everything that touches OPEN QA will be open.
Q: Can we filter on a particular commit id/branch?
A: Twice a day, the results will be pushed out. At the time of push, we can filter out some tests if needed.
Q: Do we need to scrub any of our tests for possibly sensitive info?
A: We will be sure to review the tests to ensure that there are no sensitive information included in the tests.
Q: I'm still a bit concerned with information overload from this, it looks like there are plans for having an About page but Id rather have the info more upfront (possibly left side of page?).
A: Will be working on the content in the "About" page to describe what the visitors should expect from this page. Will let you guys know once the "About" page is ready.
Q: I also had a few questions on the Open QA initiative, are there any requirements from the Community team?
A: We are currently working closely with the Community team on defining what this project should look like as moving forward.
Q: What is the goal of this side of Open QA?
A: The goal is to bring in active community engagement via transparency and complete openness inside Eucalyptus.
Q: What does the community get out of this in the short/long term?
A: In short term, this project is to display our commitment toward the openness in Eucalyptus. In long term, the benefit to build trust in quality in Eucalyptus and crowd-sourcing of Eucalyptus QA process.
Q: What is the time frame for this task?
A: We would like to get the alpha testing of the project available within the company in about 2 weeks, and get the beta testing going in about a month, being very optimistic in the timeline here.
Q: How much time should the QA team devote to it (will it impact 3.2/3.3)?
A: It will be completely up to the individual level of dedication. It should never have impact on the release of 3.2 at least.
Q: What are the goals for this task in the future?
A: The long term goal is mentioned above.
Q: When should we get engineering involved to get their input on future systems?
A: Engineering should be exposed to the project as soon as possible, prior to the alpha testing.
Q: It would be nice to begin to track this effort through Jira so we can account for the work we are doing as well as providing a release flow (ie build, test, review, push).
A: The project will be tracked once the prototype and the system architecture is agreed upon the web and IT teams.