From b8aa86bf947309a3ca9f02ce4a2837cd2ef77a72 Mon Sep 17 00:00:00 2001 From: Amar Tumballi Date: Sat, 21 Jul 2018 21:30:53 +0530 Subject: [PATCH 1/6] developer-index: fix the flow It is important to point out users that this document is pointing at 'GlusterFS', but there are more projects in gluster org. Signed-off-by: Amar Tumballi --- Developer-guide/Developers-Index.md | 34 +++++++++++++++-------------- 1 file changed, 18 insertions(+), 16 deletions(-) diff --git a/Developer-guide/Developers-Index.md b/Developer-guide/Developers-Index.md index b3f500a0..da49fa83 100644 --- a/Developer-guide/Developers-Index.md +++ b/Developer-guide/Developers-Index.md @@ -6,18 +6,20 @@ Developers Are you itching to send in patches and participate as a developer in the Gluster community? Here are a number of starting points for getting -involved. We don't require a signed contributor license agreement or -copyright assignment, but we do require a "signed-off-by" line on each -code check-in. +involved. All you need is your 'github' account to be handy. + +Remember that, [Gluster community](https://github.com/gluster) has multiple projects, each of which has its own way of handling PRs and patches. Decide on which project you want to contribute. Below document is mostly concerned about 'GlusterFS' project, which is the core of Gluster Community. Workflow -------- - [Simplified Developer Workflow](./Simplified-Development-Workflow.md) - - A simpler and faster intro to developing with GlusterFS, than the - doc below. -- [Developer Workflow](./Development-Workflow.md) - covers detail about requirements from a patch; tools and toolkits used by developers. This is recommended reading in order to begin contributions to the project. -- [GD2 Developer Workflow](https://github.com/gluster/glusterd2/blob/master/doc/development-guide.md) - Helps in on-boarding developers to contribute in GlusterD2 project. + - A simpler and faster intro to developing with GlusterFS, than the document below +- [Developer Workflow](./Development-Workflow.md) + - Covers detail about requirements from a patch; tools and toolkits used by developers. + This is recommended reading in order to begin contributions to the project. +- [GD2 Developer Workflow](https://github.com/gluster/glusterd2/blob/master/doc/development-guide.md) + - Helps in on-boarding developers to contribute in GlusterD2 project. Compiling Gluster ----------------- @@ -25,23 +27,23 @@ Compiling Gluster - [Compiling RPMS](./Compiling-RPMS.md) - Step by step instructions for compiling Gluster RPMS - [Building GlusterFS](./Building-GlusterFS.md) - How to compile - Gluster from source code. Including instructions for Ubuntu. + Gluster from source code. Developing ---------- - [Projects](./Projects.md) - Ideas for projects you could create -- [EasyFix Bugs](./Easy-Fix-Bugs.md) - Easy to fix bugs of - GlusterFS. One of the best place to start contributing to GlusterFS. - [Fixing issues reported by tools for static code analysis](./Fixing-issues-reported-by-tools-for-static-code-analysis.md) - This is a good starting point for developers to fix bugs in GlusterFS project. -- [Backport Wishlist](./Backport-Wishlist.md) - Problems fixed - in the master branch might need to get fixed in stable release - branches too. -- [Backport Guidelines](./Backport-Guidelines.md) describe the steps that - branches too. +- [EasyFix Bugs](./Easy-Fix-Bugs.md) - Easy to fix bugs of + GlusterFS. One of the best place to start contributing to GlusterFS. + +Releases and Backports +---------------------- + +- [Backport Guidelines](./Backport-Guidelines.md) describe the steps that branches too. -Other Gluster Developer documentation can be found [here](https://github.com/gluster/glusterfs/tree/master/doc/developer-guide) +Some more GlusterFS Developer documentation can be found [in glusterfs documentation directory](https://github.com/gluster/glusterfs/tree/master/doc/developer-guide) From efa2418f4d1abcf9a0e327d9bc13a8ef004e984c Mon Sep 17 00:00:00 2001 From: Amar Tumballi Date: Sat, 21 Jul 2018 23:02:41 +0530 Subject: [PATCH 2/6] Backport-Guidelines: fix the policy Clearly state what is backport, and why we do it. Remove 'Backport Wishlist' as it was out of date, and doesn't make anymore sense. Signed-off-by: Amar Tumballi --- Developer-guide/Backport-Guidelines.md | 105 ++++---------- Developer-guide/Backport-Wishlist.md | 193 ------------------------- mkdocs.yml | 1 - 3 files changed, 28 insertions(+), 271 deletions(-) delete mode 100644 Developer-guide/Backport-Wishlist.md diff --git a/Developer-guide/Backport-Guidelines.md b/Developer-guide/Backport-Guidelines.md index 198225be..f4660717 100644 --- a/Developer-guide/Backport-Guidelines.md +++ b/Developer-guide/Backport-Guidelines.md @@ -1,87 +1,38 @@ -Bugs often get fixed in master before release branches. When a bug is -fixed in the master branch, it might be desirable or necessary in a -stable branch. To put the fix in the stable branch we need to backport the -fix to the stable branch. +In GlusterFS project, as a policy, any new change, bug fix, etc., are to be +fixed in 'master' branch before release branches. When a bug is fixed in +the master branch, it might be desirable or necessary in release branch. -Anyone in the community can suggest a backport. If you are interested to -suggest a backport, please check the [Backport -Wishlist](./Backport-Wishlist.md). +This page describes the policy GlusterFS has regarding the backports. As +a user, or contributor, being aware of this policy would help you to +understand how to request for backport from community. -This page describes the steps needed to backport simple changes. Changes -that do not apply cleanly will need some manual modifications and using -`git cherry-pick` may not always be the easiest solution. +## Policy -## via Git command-line -1. Git clone the GlusterFS code +* No feature from master would be backported to the release branch +* CVE ie., security vulnerability [(listed on the CVE database)](https://cve.mitre.org/cve/search_cve_list.html) +reported in the existing releases would be backported, after getting fixed +in master branch. +* Only topics which bring about data loss or, unavailability would be +backported to the release. +* For any other issues, the project recommends that the installation be +upgraded to a newer release where the specific bug has been addressed. + - Gluster provides 'rolling' upgrade support, i.e., one can upgrade their +server version without stopping the application I/O, so we recommend migrating +to higher version. - git clone ssh://username@review.gluster.org/glusterfs +## Things to pay attention to while backporting a patch. -2. Create and checkout a new branch for your work, based on the branch - for the backport version +If your patch meets the criteria above, or you are a user, who prefer to have a +fix backported, because your current setup is facing issues, below are the +steps you need to take care to submit a patch on release branch. - git checkout -t -b bug-123456/release-3.8 origin/release-3.8 +* The patch should have same 'Change-Id'. +* The bugzilla ID should be for the same release branch. -3. Cherry pick the change from the master branch. - $ git cherry-pick -x a0b1c2d3e4f5 - - verify that the change has been merged into the master branch. +### How to contact release owners? -4. Update/correct the commit message. +All release owners are part of 'gluster-devel@gluster.org' mailing list. +Please write your expectation from next release there, so we can take that +to consideration while making the release. - $ git commit -s --amend --date="$(date)" -[This is one example](https://github.com/gluster/glusterfs/commit/40407afb529f6e5fa2f79e9778c2f527122d75eb) of the commit message that has a good description for a backport. Notice the indention of the patch-metadata like BUG, Change-ID and Reviewed-on tags. There is also the original commit-id that was cherry-picked from the master branch. - -make sure to quote the review tags - -update the BUG reference, point to the BUG that is used for this -particular release-branch - -add a Signed-off-by tag - -5. Run `./rfc.sh` to post the backport for review. - - ./rfc.sh - -## via Gerrit web interface -1. Navigate to the required change in Gerrit from any web browser. - -2. Click on 'Cherry Pick' button. You will now be presented with a dialogue box with two editable fields. First - one is to specify the branch to which this particular change needs to be -cherry-picked and second one for modifying the already existing commit message. - -3. Start entering the branch name in the corresponding field and you can select - the required branch from the list of available branches. - -4. Make sure that you only edit the following from the commit message: - - * BUG: Replace with the correct bug-id reported against the branch to which change is going to be backported. - * Prefix all other lines except the commit message, Signed-off-by, cherry-picked and Change-Id lines with a greater than symbol and a whitespace '> ' and re-arrange as a whole to have the following format: - - - . . . - - - > Reviewed-on: http://review.gluster.org/ - > Smoke: Gluster Build System - > CentOS-regression: Gluster Build System - > NetBSD-regression: NetBSD Build System - > Reviewed-by: username1 - > Reviewed-by: username2 - > Reviewed-by: username3 - - (cherry picked from commit ) - - Change-Id: - BUG: - Signed-off-by: username - -5. Click on 'Cherry Pick Change'. You will now be re-directed to the new change URL. - -6. Click on the edit button adjacent to the field named 'Topic'. - -7. Add the following as 'Topic': - - bug- - Note:- Replace with the required bug-id for that branch. - -8. Click on 'Update' or 'Submit'. - -After submitting the patch(es), make sure to move the bug to the *POST* -status. diff --git a/Developer-guide/Backport-Wishlist.md b/Developer-guide/Backport-Wishlist.md deleted file mode 100644 index f191820c..00000000 --- a/Developer-guide/Backport-Wishlist.md +++ /dev/null @@ -1,193 +0,0 @@ -Bugs often get fixed in master before release branches. - -When a bug is fixed in the master branch it might be desirable or -necessary to backport the fix to a stable branch. - -This page is intended to help organize support (and prioritization) for -backporting bug fixes of importance to the community. - -### GlusterFs 3.6 - -Requested Backports for 3.6.0 ------------------------------ - -The tracker bug for 3.6.0 : - - -Please add 'glusterfs-3.6.0' in the 'Blocks' field of bugs to propose -inclusion in GlusterFS 3.6.0. - -### GlusterFs 3.5 - -Requested Backports for 3.5.3 ------------------------------ - -Current [list of bugs planned for -inclusion](https://bugzilla.redhat.com/showdependencytree.cgi?hide_resolved=0&id=glusterfs-3.5.3). - -- File a new bug for backporting a patch to 3.5.3: - [... - new glusterfs-3.5.3 backport request] - -### GlusterFs 3.4 - -Requested Backports for 3.4.6 ------------------------------ - -The tracker bug for 3.4.6 : - - -Please add 'glusterfs-3.4.6' in the 'Blocks' field of bugs to propose -inclusion in GlusterFS 3.4.6. - - - - -Requested Backports for 3.4.4 ------------------------------ - - - "self-heal -process can sometimes create directories instead of symlinks for the -root gfid file in .glusterfs" - - - "structure needs -cleaning" message appear when accessing files. - - - glusterfs mount -crash after remove brick, detach peer and termination - -Requested Backports for 3.4.3 ------------------------------ - - - "self-heal -process can sometimes create directories instead of symlinks for the -root gfid file in .glusterfs" - - - "structure needs -cleaning" message appear when accessing files. - - - large NFS writes -to Gluster slow down then stop - - - glusterfs mount -crash after remove brick, detach peer and termination - -Requested Backports for 3.3.3 ------------------------------ - -[Enable fusermount by default, make nightly autobuilding -work](https://bugzilla.redhat.com/1058666) - -Requested Backports for 3.4.2 ------------------------------ - -Please enter bugzilla ID or patch URL here: - -​1) Until RDMA handling is improved, we should output a warning when -using RDMA volumes - - - -​2) Unable to shrink volumes without dataloss - - - -​3) cluster/dht: Allow non-local clients to function with nufa volumes. -- - -Requested Backports for 3.4.1 ------------------------------ - -Please enter bugzilla ID or patch URL here. - - - "quota context -not set in inode" - - - "NFS crash bug" - -A note for whoever reviews this list: These are the fixes for issues -that have caused actual service disruption in our production -installation and thus are absolutely required for us (-- Lubomir -Rintel): - - - "Setting ACL -entries fails with glusterfs-3.4.0" - - - "fd leaks -observed while running dbench with "open-behind" volume option set to -"on" on a replicate volume" - -These are issues that we've stumbled upon during the git log review and -that seemed scary enough for us to cherry-pick them to avoid risk, -despite not being actually hit. Hope that helps deciding whether it's -worthwhile cherry-picking them (-- Lubomir Rintel): - - "CLI crash upon -executing "gluster peer status" command" - - "quick-read and -open-behind xlator: Make options (volume\_options ) structure NULL -terminated." - - "nfs-root-squash: -rename creates a file on a file residing inside a sticky bit set -directory" - - "DHT : files are -stored on directory which doesn't have hash range(hash layout)" - - "statedump crashes -in ioc\_inode\_dump" - - "cli crashes when -setting diagnostics.client-log-level is set to trace" - - "glusterfsd crashes -on smallfile benchmark" - -, "tests: call 'cleanup' at the end of -each test", , -backport of 983975 - -, "glusterfs-api.pc.in contains an -rpath", , backport -of 1002220 - - "glusterd.service (systemd), ensure -glusterd starts before any local gluster mounts", -, backport of -1004795 - - meta, check that -glusterfs.spec.in has all relevant updates - - - Glusterd would -not store all the volumes when a global options were set leading to peer -rejection - -Requested Backports -------------------- - -- Please backport [gfapi: Closed the logfile fd and initialize to NULL - in glfs\_fini](http://review.gluster.org/#/c/6552) into release-3.5 - - Done -- Please backport [cluster/dht: Make sure loc has - gfid](http://review.gluster.org/5178) into release-3.4 -- Please backport [Bug 887098](http://goo.gl/QjeMP) into release-3.3 - (FyreFoX) - Done -- Please backport [Bug 856341](http://goo.gl/9cGAC) into release-3.2 - and release-3.3 (the-me o/b/o Debian) - Done for release-3.3 -- Please backport [Bug 895656](http://goo.gl/ZNs3J) into release-3.2 - and release-3.3 (semiosis, x4rlos) - Done for release-3.3 -- Please backport [Bug 918437](http://goo.gl/1QRyw) into release-3.3 - (tjstansell) - Done -- Please backport into [Bug - 884597](https://bugzilla.redhat.com/show_bug.cgi?id=884597) - release-3.3 (nocko) - Done - -Unaddressed bugs ----------------- - -- [Bug 838784](https://bugzilla.redhat.com/show_bug.cgi?id=838784) -- [Bug 893778](https://bugzilla.redhat.com/show_bug.cgi?id=893778) -- [Bug 913699](https://bugzilla.redhat.com/show_bug.cgi?id=913699); - possibly related to [Bug - 884597](https://bugzilla.redhat.com/show_bug.cgi?id=884597) \ No newline at end of file diff --git a/mkdocs.yml b/mkdocs.yml index 82f3c704..6be23a38 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -79,7 +79,6 @@ pages: - Project Ideas : Developer-guide/Projects.md - EasyFix bugs : Developer-guide/Easy-Fix-Bugs.md - Fixing issues reported by tools for static code analysis : Developer-guide/Fixing-issues-reported-by-tools-for-static-code-analysis.md - - Backport Wishlist : Developer-guide/Backport-Wishlist.md - Backport Guidelines : Developer-guide/Backport-Guidelines.md - Contributors Guide: - Index: Contributors-Guide/Index.md From 01b2a58311abb02bb2282f4380018f8474039cf6 Mon Sep 17 00:00:00 2001 From: Amar Tumballi Date: Sat, 21 Jul 2018 23:09:46 +0530 Subject: [PATCH 3/6] Coverity: minor fixes * Remove Easy-Fix list of bugs from bugzilla. * Coverity document is revisited with minor changes Signed-off-by: Amar Tumballi --- Developer-guide/Developers-Index.md | 2 -- ...sues-reported-by-tools-for-static-code-analysis.md | 11 ++--------- 2 files changed, 2 insertions(+), 11 deletions(-) diff --git a/Developer-guide/Developers-Index.md b/Developer-guide/Developers-Index.md index da49fa83..9827a0ec 100644 --- a/Developer-guide/Developers-Index.md +++ b/Developer-guide/Developers-Index.md @@ -38,8 +38,6 @@ Developing analysis](./Fixing-issues-reported-by-tools-for-static-code-analysis.md) - This is a good starting point for developers to fix bugs in GlusterFS project. -- [EasyFix Bugs](./Easy-Fix-Bugs.md) - Easy to fix bugs of - GlusterFS. One of the best place to start contributing to GlusterFS. Releases and Backports ---------------------- diff --git a/Developer-guide/Fixing-issues-reported-by-tools-for-static-code-analysis.md b/Developer-guide/Fixing-issues-reported-by-tools-for-static-code-analysis.md index 66f412e8..6e792626 100644 --- a/Developer-guide/Fixing-issues-reported-by-tools-for-static-code-analysis.md +++ b/Developer-guide/Fixing-issues-reported-by-tools-for-static-code-analysis.md @@ -19,12 +19,8 @@ program. - Once admins for the GlusterFS Coverity scan approve your request, you will be able to see the defects raised by Coverity. - [BZ 789278](https://bugzilla.redhat.com/show_bug.cgi?id=789278) - should be used as a umbrella bug for Coverity issues in master + can be used as a umbrella bug for Coverity issues in master branch unless you are trying to fix a specific bug in Bugzilla. - - While sending patches for fixing Coverity issues please use the - same bug number. - - For 3.6 branch the Coverity tracking bug is - [1122834](https://bugzilla.redhat.com/show_bug.cgi?id=1122834) - When you decide to work on some issue, please assign it to your name in the same Coverity website. So that we don't step on each others work. @@ -42,7 +38,7 @@ Cppcheck is available in Fedora and EL's EPEL repo - Install Cppcheck - yum install cppcheck + dnf install cppcheck - Clone GlusterFS code @@ -52,9 +48,6 @@ Cppcheck is available in Fedora and EL's EPEL repo cppcheck glusterfs/ 2>cppcheck.log -- [BZ 1091677](https://bugzilla.redhat.com/show_bug.cgi?id=1091677) - should be used for submitting patches to master branch for Cppcheck - reported issues. ### Daily Runs From 99571392e1f9632dd14755968425b9b0b04d6ddc Mon Sep 17 00:00:00 2001 From: Amar Tumballi Date: Sat, 21 Jul 2018 23:19:29 +0530 Subject: [PATCH 4/6] Projects: cleanup few things Signed-off-by: Amar Tumballi --- Developer-guide/Projects.md | 56 +++++++++++++++++++++---------------- 1 file changed, 32 insertions(+), 24 deletions(-) diff --git a/Developer-guide/Projects.md b/Developer-guide/Projects.md index 5c41fef9..a9fb7b4d 100644 --- a/Developer-guide/Projects.md +++ b/Developer-guide/Projects.md @@ -1,28 +1,44 @@ This page contains a list of project ideas which will be suitable for students (for GSOC, internship etc.) -Projects with mentors ---------------------- +Projects/Features which needs contributors +------------------------------------------ -### gfsck - A GlusterFS filesystem check +### glusterd2 -- A tool to check filesystem integrity and repairing -- I'm currently working on it -- Owner: Xavier Hernandez (Datalab) +Repo: https://github.com/gluster/glusterd2 + +New management layer for glusterfs. Has native ReST API, and uses etcd to +store the data, and hence would solve more common GlusterD scale issues seen today. + +### RIO -### Sub-directory mount support for native GlusterFS mounts +Issue: https://github.com/gluster/glusterfs/issues/243 -Allow clients to directly mount directories inside a GlusterFS volume, -like how NFS clients can mount directories inside an NFS export. +This is a new distribution logic, which can scale Gluster to 1000s of nodes. -Mentor: Kaushal +### gluster-block -### GlusterD services high availablity +Repo: https://github.com/gluster/gluster-block -GlusterD should restart the processes it manages, bricks, nfs server, -self-heal daemon and quota daemon, whenever it detects they have died. +The project tries to expose files in glusterfs as iSCSI block devices. A very good +way to get the transaction workload work on GlusterFS. -Mentor : Atin Mukherjee +### GlusterFS UFO/SWIFT + +To build and run Gluster UFO you can do the following: + +1. Configure UFO/SWIFT as described in [Howto Using UFO SWIFT - A quick + and dirty setup + guide](https://github.com/gluster/gluster-swift/blob/master/doc/markdown/quick_start_guide.md) + + +Projects with mentors +--------------------- + +### gfsck - A GlusterFS filesystem check + +- A tool to check filesystem integrity and repairing ### Language bindings for libgfapi @@ -62,22 +78,14 @@ be a special rebalance process. Improve rebalance performance. -### Meta translator - -The meta xlator provides a /proc like interface to GlusterFS xlators. -This could be improved upon and the meta xlator could be made a standard -part of the volume graph. - -### Geo-replication using rest-api - -Might be suitable for geo replication over WAN. - ### Quota using underlying FS' quota GlusterFS quota is currently maintained completely in GlusterFSs namespace using xattrs. We could make use of the quota capabilities of the underlying fs (XFS) for better performance. +Check [#184](https://github.com/gluster/glusterfs/184) + ### Snapshot pluggability Snapshot should be able to make use of snapshot support provided by From 491e3bc6a7e64eb06e5894631099688627b9176b Mon Sep 17 00:00:00 2001 From: Amar Tumballi Date: Sat, 21 Jul 2018 23:25:13 +0530 Subject: [PATCH 5/6] Compiling-RPMs: remove it from documentation Major reason for removing is, it is out-of-date. Second, we don't expect users to build RPMs, they can consume the nightly RPM builds from our CI systems. For every release, we do build and provide RPMs. For all other purposes, source install is sufficient. Signed-off-by: Amar Tumballi --- Developer-guide/Building-GlusterFS.md | 5 +- Developer-guide/Compiling-RPMS.md | 178 ------------------------ Developer-guide/Developers-Index.md | 2 - Developer-guide/Development-Workflow.md | 22 +-- mkdocs.yml | 1 - 5 files changed, 3 insertions(+), 205 deletions(-) delete mode 100644 Developer-guide/Compiling-RPMS.md diff --git a/Developer-guide/Building-GlusterFS.md b/Developer-guide/Building-GlusterFS.md index 76b750bc..f2fd7c62 100644 --- a/Developer-guide/Building-GlusterFS.md +++ b/Developer-guide/Building-GlusterFS.md @@ -31,7 +31,7 @@ The following packages are required for building GlusterFS, The following dnf command installs all the build requirements for Fedora, - # dnf install automake autoconf libtool flex bison openssl-devel libxml2-devel python-devel libaio-devel libibverbs-devel librdmacm-devel readline-devel lvm2-devel glib2-devel userspace-rcu-devel libcmocka-devel libacl-devel sqlite-devel fuse-devel redhat-rpm-config + # dnf install automake autoconf libtool flex bison openssl-devel libxml2-devel python-devel libaio-devel libibverbs-devel librdmacm-devel readline-devel lvm2-devel glib2-devel userspace-rcu-devel libcmocka-devel libacl-devel sqlite-devel fuse-devel redhat-rpm-config rpcgen libtirpc-devel make ### Ubuntu @@ -169,6 +169,3 @@ steps to build RPMs, This will create rpms from the source in 'extras/LinuxRPM'. *(Note: You will need to install the rpmbuild requirements including rpmbuild and mock)* - -A more detailed description for building RPMs can be found at -[CompilingRPMS](./Compiling-RPMS.md). diff --git a/Developer-guide/Compiling-RPMS.md b/Developer-guide/Compiling-RPMS.md deleted file mode 100644 index 819985da..00000000 --- a/Developer-guide/Compiling-RPMS.md +++ /dev/null @@ -1,178 +0,0 @@ -How to compile GlusterFS RPMs from git source, for RHEL/CentOS, and Fedora --------------------------------------------------------------------------- - -Creating rpm's of GlusterFS from git source is fairly easy, once you -know the steps. - -RPMS can be compiled on at least the following OS's: - -- Red Hat Enterprise Linux 5, 6 (& 7 when available) -- CentOS 5, 6 (& 7 when available) -- Fedora 16-20 - -Specific instructions for compiling are below. If you're using: - -- Fedora 16-20 - Follow the Fedora steps, then do all of the Common - steps. -- CentOS 5.x - Follow the CentOS 5.x steps, then do all of the Common - steps -- CentOS 6.x - Follow the CentOS 6.x steps, then do all of the Common - steps. -- RHEL 6.x - Follow the RHEL 6.x steps, then do all of the Common - steps. - -Note - these instructions have been explicitly tested on all of CentOS -5.10, RHEL 6.4, CentOS 6.4+, and Fedora 16-20. Other releases of -RHEL/CentOS and Fedora may work too, but haven't been tested. Please -update this page appropriately if you do so. :) - -### Preparation steps for Fedora 16-20 (only) - -​1. Install gcc, the python development headers, and python setuptools: - - $ sudo yum -y install gcc python-devel python-setuptools - -​2. If you're compiling GlusterFS version 3.4, then install -python-swiftclient. Other GlusterFS versions don't need it: - - $ sudo easy_install simplejson python-swiftclient - -Now follow through the **Common Steps** part below. - -### Preparation steps for CentOS 5.x (only) - -You'll need EPEL installed first and some CentOS specific packages. The -commands below will get that done for you. After that, follow through -the "Common steps" section. - -​1. Install EPEL first: - - $ curl -OL https://download.fedoraproject.org/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm - $ sudo yum -y install epel-release-5-4.noarch.rpm - -​2. Install the packages required only on CentOS 5.x: - - $ sudo yum -y install buildsys-macros gcc ncurses-devel python-ctypes python-sphinx10 \ -   redhat-rpm-config - -Now follow through the **Common Steps** part below. - -### Preparation steps for CentOS 6.x (only) - -You'll need EPEL installed first and some CentOS specific packages. The -commands below will get that done for you. After that, follow through -the "Common steps" section. - -​1. Install EPEL first: - - $ sudo yum -y install https://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm - -​2. Install the packages required only on CentOS: - - $ sudo yum -y install python-webob1.0 python-paste-deploy1.5 python-sphinx10 redhat-rpm-config - -Now follow through the **Common Steps** part below. - -### Preparation steps for RHEL 6.x (only) - -You'll need EPEL installed first and some RHEL specific packages. The 2 -commands below will get that done for you. After that, follow through -the "Common steps" section. - -​1. Install EPEL first: - - $ sudo yum -y install https://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm - -​2. Install the packages required only on RHEL: - - $ sudo yum -y --enablerepo=rhel-6-server-optional-rpms install python-webob1.0 \ -   python-paste-deploy1.5 python-sphinx10 redhat-rpm-config - -Now follow through the **Common Steps** part below. - -### Common Steps - -These steps are for both Fedora and RHEL/CentOS. At the end you'll have -the complete set of GlusterFS RPMs for your platform, ready to be -installed. - -**NOTES for step 1 below:** - -- If you're on RHEL/CentOS 5.x and get a message about lvm2-devel not - being available, it's ok. You can ignore it. :) -- If you're on RHEL/CentOS 6.x and get any messages about - python-eventlet, python-netifaces, python-sphinx and/or pyxattr not - being available, it's ok. You can ignore them. :) - -​1. Install the needed packages - - $ sudo yum -y --disablerepo=rhs* --enablerepo=*optional-rpms install git autoconf \ -   automake bison cmockery2-devel dos2unix flex fuse-devel glib2-devel libaio-devel \ -   libattr-devel libibverbs-devel librdmacm-devel libtool libxml2-devel lvm2-devel make \ -   openssl-devel pkgconfig pyliblzma python-devel python-eventlet python-netifaces \ -   python-paste-deploy python-simplejson python-sphinx python-webob pyxattr readline-devel \ -   rpm-build systemtap-sdt-devel tar libcmocka-devel - -​2. Clone the GlusterFS git repository - - $ git clone https://review.gluster.org/glusterfs - $ cd glusterfs - -​3. Choose which branch to compile - -If you want to compile the latest development code, you can skip this -step and go on to the next one. - -If instead you want to compile the code for a specific release of -GlusterFS (such as v3.4), get the list of release names here: - - $ git branch -a | grep release -   remotes/origin/release-2.0 -   remotes/origin/release-3.0 -   remotes/origin/release-3.1 -   remotes/origin/release-3.2 -   remotes/origin/release-3.3 -   remotes/origin/release-3.4 -   remotes/origin/release-3.5 - -Then switch to the correct release using the git "checkout" command, and -the name of the release after the "remotes/origin/" bit from the list -above: - - $ git checkout release-3.4 - -**NOTE -** The CentOS 5.x instructions have only been tested for the -master branch in GlusterFS git. It is unknown (yet) if they work for -branches older then release-3.5. - -​4. Configure and compile GlusterFS - -Now you're ready to compile Gluster: - - $ ./autogen.sh - $ ./configure --enable-fusermount - $ make dist - -​5. Create the GlusterFS RPMs - - $ cd extras/LinuxRPM - $ make glusterrpms - -That should complete with no errors, leaving you with a directory -containing the RPMs. - - $ ls -l *rpm - -rw-rw-r-- 1 jc jc 3966111 Mar  2 12:15 glusterfs-3git-1.el5.centos.src.rpm - -rw-rw-r-- 1 jc jc 1548890 Mar  2 12:17 glusterfs-3git-1.el5.centos.x86_64.rpm - -rw-rw-r-- 1 jc jc   66680 Mar  2 12:17 glusterfs-api-3git-1.el5.centos.x86_64.rpm - -rw-rw-r-- 1 jc jc   20399 Mar  2 12:17 glusterfs-api-devel-3git-1.el5.centos.x86_64.rpm - -rw-rw-r-- 1 jc jc  123806 Mar  2 12:17 glusterfs-cli-3git-1.el5.centos.x86_64.rpm - -rw-rw-r-- 1 jc jc 7850357 Mar  2 12:17 glusterfs-debuginfo-3git-1.el5.centos.x86_64.rpm - -rw-rw-r-- 1 jc jc  112677 Mar  2 12:17 glusterfs-devel-3git-1.el5.centos.x86_64.rpm - -rw-rw-r-- 1 jc jc  100410 Mar  2 12:17 glusterfs-fuse-3git-1.el5.centos.x86_64.rpm - -rw-rw-r-- 1 jc jc  187221 Mar  2 12:17 glusterfs-geo-replication-3git-1.el5.centos.x86_64.rpm - -rw-rw-r-- 1 jc jc  299171 Mar  2 12:17 glusterfs-libs-3git-1.el5.centos.x86_64.rpm - -rw-rw-r-- 1 jc jc   44943 Mar  2 12:17 glusterfs-rdma-3git-1.el5.centos.x86_64.rpm - -rw-rw-r-- 1 jc jc  123065 Mar  2 12:17 glusterfs-regression-tests-3git-1.el5.centos.x86_64.rpm - -rw-rw-r-- 1 jc jc   16224 Mar  2 12:17 glusterfs-resource-agents-3git-1.el5.centos.x86_64.rpm - -rw-rw-r-- 1 jc jc  654043 Mar  2 12:17 glusterfs-server-3git-1.el5.centos.x86_64.rpm diff --git a/Developer-guide/Developers-Index.md b/Developer-guide/Developers-Index.md index 9827a0ec..d507df6e 100644 --- a/Developer-guide/Developers-Index.md +++ b/Developer-guide/Developers-Index.md @@ -24,8 +24,6 @@ Workflow Compiling Gluster ----------------- -- [Compiling RPMS](./Compiling-RPMS.md) - Step by step - instructions for compiling Gluster RPMS - [Building GlusterFS](./Building-GlusterFS.md) - How to compile Gluster from source code. diff --git a/Developer-guide/Development-Workflow.md b/Developer-guide/Development-Workflow.md index 264fe14a..4c271f9e 100644 --- a/Developer-guide/Development-Workflow.md +++ b/Developer-guide/Development-Workflow.md @@ -170,11 +170,6 @@ command to install the required packages: pkg-config python-dev python-eventlet python-netifaces libaio-dev libibverbs-dev libtool libxml2-dev liblvm2-dev make autoconf automake bison dos2unix flex libfuse-dev -CentOS/RHEL/Fedora: - -On Fedora systems, install the required packages by following the -instructions in [CompilingRPMS](./Compiling-RPMS.md). - ### Creating build environment Once the required packages are installed for your appropiate system, @@ -187,26 +182,13 @@ generate the build configuration: #### GlusterFS -Ubuntu: - -Type the following to build and install GlusterFS on the system: - - sh$ make - sh$ make install - -CentOS/RHEL/Fedora: - -In an rpm based system, there are two methods to build GlusterFS. One is -to use the method describe above for *Ubuntu*. The other is to build and -install RPMS as described in [CompilingRPMS](./Compiling-RPMS.md). + sh$ make install #### GlusterFS UFO/SWIFT To build and run Gluster UFO you can do the following: -1. Build, create, and install the RPMS as described in - [CompilingRPMS](./Compiling-RPMS.md). -2. Configure UFO/SWIFT as described in [Howto Using UFO SWIFT - A quick +1. Configure UFO/SWIFT as described in [Howto Using UFO SWIFT - A quick and dirty setup guide](http://www.gluster.org/2012/09/howto-using-ufo-swift-a-quick-and-dirty-setup-guide) diff --git a/mkdocs.yml b/mkdocs.yml index 6be23a38..7232716f 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -74,7 +74,6 @@ pages: - Developers Home: Developer-guide/Developers-Index.md - Simplified Development Workflow : Developer-guide/Simplified-Development-Workflow.md - Development-Workflow : Developer-guide/Development-Workflow.md - - Compiling RPMS from GIT Source : Developer-guide/Compiling-RPMS.md - Build and Install GlusterFS : Developer-guide/Building-GlusterFS.md - Project Ideas : Developer-guide/Projects.md - EasyFix bugs : Developer-guide/Easy-Fix-Bugs.md From 936edb06adf9719dfc0a2bf930099d506b3cfad1 Mon Sep 17 00:00:00 2001 From: Amar Tumballi Date: Sat, 21 Jul 2018 23:36:22 +0530 Subject: [PATCH 6/6] Development Workflow: bring some sanity Update the document to point to the latest facts about the contributions, so one needn't get confused. Signed-off-by: Amar Tumballi --- Developer-guide/Development-Workflow.md | 98 +++++++------------------ 1 file changed, 28 insertions(+), 70 deletions(-) diff --git a/Developer-guide/Development-Workflow.md b/Developer-guide/Development-Workflow.md index 4c271f9e..fa7c8f22 100644 --- a/Developer-guide/Development-Workflow.md +++ b/Developer-guide/Development-Workflow.md @@ -60,9 +60,9 @@ code. ### Register -Sign up for an account at by clicking -'Register' on the right-hand top. You can use your gmail login as the -openID identity. +You need to have a github.com account to register for 'review.gluster.org'. +Once you open https://review.gluster.org, click on 'Register' button, and +use your github account to register. ### Preferred email @@ -116,12 +116,12 @@ This is where new features come in first. In fact this is where almost every change (commit) comes in first. The master branch is always kept in a buildable state and smoke tests pass. -Release trains (3.1.z, 3.2.z, 3.2.z) each have a branch originating from +Release trains (3.1.z, 3.2.z, 4.y, 5.y) each have a branch originating from master. Code freeze of each new release train is marked by the creation -of the release-3.y branch. At this point no new features are added to -the release-3.y branch. All fixes and commits first get into master. +of the `release-x.y` branch. At this point no new features are added to +the release-x.y branch. All fixes and commits first get into master. From there, only bug fixes get backported to the relevant release -branches. From the release-3.y branch, actual release code snapshots +branches. From the release-x.y branch, actual release code snapshots (e.g. glusterfs-3.2.2 etc.) are tagged (git annotated tag with 'git tag -a') shipped as a tarball. @@ -137,21 +137,6 @@ local task branch like this - sh$ git branch bug-XYZ && git checkout bug-XYZ ...  -If you are backporting a fix to a release branch, or making a new change -to a release branch, your commands would be slightly different. If you -are checking out a release branch in your local working tree for the -first time, make sure to set it up as a remote tracking branch like this -- - - sh$ git checkout -b release-3.2 origin/release-3.2 - -The above step is not necessary to be repeated. In the future if you -want to work to the release branch - - - sh$ git checkout release-3.2 - sh$ git branch bug-XYZ-release-3.2 && git checkout bug-XYZ-release-3.2 - ...  - Building -------- @@ -160,16 +145,6 @@ Building **For details about the required packages for the build environment refer : [Building GlusterFS](./Building-GlusterFS.md)** -Ubuntu: - -To setup the build environment on an Ubuntu system, type the following -command to install the required packages: - - sudo apt-get -y install python-pyxattr libreadline-dev systemtap-sdt-dev - tar python-pastedeploy python-simplejson python-sphinx python-webob libssl-dev - pkg-config python-dev python-eventlet python-netifaces libaio-dev libibverbs-dev - libtool libxml2-dev liblvm2-dev make autoconf automake bison dos2unix flex libfuse-dev - ### Creating build environment Once the required packages are installed for your appropiate system, @@ -184,14 +159,6 @@ generate the build configuration: sh$ make install -#### GlusterFS UFO/SWIFT - -To build and run Gluster UFO you can do the following: - -1. Configure UFO/SWIFT as described in [Howto Using UFO SWIFT - A quick - and dirty setup - guide](http://www.gluster.org/2012/09/howto-using-ufo-swift-a-quick-and-dirty-setup-guide) - Commit policy ------------- @@ -199,16 +166,6 @@ For a Gerrit based work flow, each commit should be an independent, buildable and testable change. Typically you would have a local branch per task, and most of the times that branch will have one commit. -If you have a second task at hand which depends on the changes of the -first one, then technically you can have it as a separate commit on top -of the first commit. But it is important that the first commit should be -a testable change by itself (if not, it is an indication that the two -commits are essentially part of a single change). Gerrit accommodates -these situations by marking Change 1 as a "dependency" of Change 2 -(there is a 'Dependencies' tab in the Change page in Gerrit) -automatically when you push the changes for review from the same local -branch. - You will need to sign-off your commit (git commit -s) before sending the patch for review. By signing off your patch, you agree to the terms listed under "Developer's Certificate of Origin" section in the @@ -223,32 +180,22 @@ the following format - Description of the code changes - Reason for doing it this way (compared to others) - Description of test cases +- A reference ID ### Test cases Part of the workflow is to aggregate and execute pre-commit test cases which accompany patches, cumulatively for every new patch. This guarantees that tests which are working till the present are not broken -with the new patch. Every change submitted to Gerrit much include test -cases in - - tests/group/script.t - -as part of the patch. This is so that code changes and accompanying test -cases are reviewed together. All new commits now come under the -following categories w.r.t test cases: +with the new patch. This is so that code changes and accompanying test +cases are reviewed together. For any new feature that is posted for review, there should be accompanying set of tests in -[distaf](https://github.com/gluster/distaf/blob/master/README.md). These +[glusto-tests](https://github.com/gluster/glusto-tests). These tests will be run nightly and/or before release to determine the health -of the feature. Please read the -[HOWTO](https://github.com/gluster/distaf/blob/master/docs/HOWTO.md) for -more information on how to write and execute the tests in distaf. - -#### New 'group' directory and/or 'script.t' - -This is typically when code is adding a new module and/or feature +of the feature. Please go through glusto-tests project to understand +more information on how to write and execute the tests in glusto. #### Extend/Modify old test cases in existing scripts @@ -300,23 +247,26 @@ This script does the following: On a successful push, you will see a URL pointing to the change in review.gluster.org -Auto verification ------------------ +Verification +------------ The integration between Jenkins and Gerrit triggers an event in Jenkins on every push of changes, to pick up the change and run build and smoke test on it. If the build and smoke tests execute successfully, Jenkins marks the -change as '+0 Verified'. If they fail, '-1 Verified' is marked on the +change as '+0 Smoke'. If they fail, '-1 Smoke' is marked on the change. This means passing the automated smoke test is a necessary condition but not sufficient. +Currently marking 'Verified' flag is manual, and should be done once +the 'smoke' tests pass. Once this flag is set, the elaborate regression tests will start running. + It is important to note that Jenkins verification is only a generic verification of high level tests. More concentrated testing effort for the patch is necessary with manual verification. -If auto verification fails, it is a good reason to skip code review till +If regression fails, it is a good reason to skip code review till a fixed change is pushed later. You can click on the build URL automatically put as a comment to inspect the reason for auto verification failure. In the Jenkins job page, you can click on the @@ -385,6 +335,11 @@ in the review/resubmit cycle. Instead it is triggered by the maintainers, after code review. Passing the regression test is a necessary condition for merge along with code review points. +To run all regession tests locally, run below script from glusterfs root directory. + + sh$ ./run-tests.sh + + Submission Qualifiers --------------------- @@ -394,6 +349,9 @@ Reviewed', and a change should have at least one '+1 Verified' (regression test). The project maintainer will merge the changes once a patch meets these qualifiers. +NOTE: The regression tests gets triggered only after marking Verified +as +1 to your patch, which later votes for 'Regression' field. + Submission Disqualifiers ------------------------