Merge pull request #10103 from IQSS/10101-qa-guide

stub out QA guide

doc/release-notes/10101-qa-guide.md

@@ -0,0 +1 @@
+A new QA Guide is intended mostly for the core development team but may be of interest to contributors.

doc/sphinx-guides/requirements.txt

@@ -1,12 +1,7 @@
-# Developers, please use Python 3.9 or lower to build the guides.
-# For your convenience, a solution for Python 3.10 is provided below
-# but we would prefer that you use the same version of Sphinx
-# (below on the < 3.10 line) that is used to build the production guides.
-Sphinx==3.5.4 ; python_version < '3.10'
-Sphinx==5.3.0 ; python_version >= '3.10'
+Sphinx==7.2.6
 
-# Necessary workaround for ReadTheDocs for Sphinx 3.x - unnecessary as of Sphinx 4.5+
-Jinja2>=3.0.2,<3.1
-
-# Sphinx - Additional modules
+# inline icons
 sphinx-icon==0.1.2
+
+# Markdown support
+myst-parser==2.0.0

doc/sphinx-guides/source/conf.py

@@ -38,11 +38,11 @@
 # ones.
 extensions = [
     'sphinx.ext.autodoc',
-    'sphinx.ext.intersphinx',
     'sphinx.ext.ifconfig',
     'sphinx.ext.viewcode',
     'sphinx.ext.graphviz',
     'sphinxcontrib.icon',
+    'myst_parser',
 ]
 
 # Add any paths that contain templates here, relative to this directory.
@@ -430,9 +430,6 @@
 # If false, no index is generated.
 #epub_use_index = True
 
-
-# Example configuration for intersphinx: refer to the Python standard library.
-intersphinx_mapping = {'https://docs.python.org/': None}
 # Suppress "WARNING: unknown mimetype for ..." https://github.com/IQSS/dataverse/issues/3391
 suppress_warnings = ['epub.unknown_project_files']
 rst_prolog = """

doc/sphinx-guides/source/developers/intro.rst

@@ -37,6 +37,8 @@ Roadmap
 
 For the Dataverse Software development roadmap, please see https://www.iq.harvard.edu/roadmap-dataverse-project
 
+.. _kanban-board:
+
 Kanban Board
 ------------
 

doc/sphinx-guides/source/developers/making-releases.rst

@@ -83,6 +83,8 @@ To test these images against our API test suite, go to the "alpha" workflow at h
 
 If there are failures, additional dependencies or settings may have been added to the "develop" workflow. Copy them over and try again.
 
+.. _build-guides:
+
 Build the Guides for the Release
 --------------------------------
 

doc/sphinx-guides/source/developers/sql-upgrade-scripts.rst

@@ -21,6 +21,8 @@ If you are creating a new database table (which maps to an ``@Entity`` in JPA),
 
 If you are doing anything other than creating a new database table such as adding a column to an existing table, you must create or update a SQL upgrade script.
 
+.. _create-sql-script:
+
 How to Create a SQL Upgrade Script
 ----------------------------------
 

doc/sphinx-guides/source/developers/testing.rst

@@ -427,6 +427,8 @@ target/coverage-it/index.html is the place to start reading the code coverage re
 Load/Performance Testing
 ------------------------
 
+See also :doc:`/qa/performance-tests` in the QA Guide.
+
 .. _locust:
 
 Locust

doc/sphinx-guides/source/developers/version-control.rst

@@ -34,6 +34,8 @@ The "master" Branch
 
 The "`master <https://github.com/IQSS/dataverse/tree/master>`_" branch represents released versions of the Dataverse Software. As mentioned in the :doc:`making-releases` section, at release time we update the master branch to include all the code for that release. Commits are never made directly to master. Rather, master is updated only when we merge code into it from the "develop" branch.
 
+.. _develop-branch:
+
 The "develop" Branch
 ********************
 

doc/sphinx-guides/source/index.rst

@@ -20,6 +20,7 @@ These documentation guides are for the |version| version of Dataverse. To find g
   developers/index
   container/index
   style/index
+  qa/index.md
 
 How the Guides Are Organized
 ----------------------------

doc/sphinx-guides/source/qa/index.md

@@ -0,0 +1,10 @@
+# QA Guide
+
+```{toctree}
+overview.md
+testing-approach.md
+testing-infrastructure.md
+qa-workflow.md
+test-automation.md
+performance-tests.md
+```

doc/sphinx-guides/source/qa/overview.md

@@ -0,0 +1,63 @@
+# Overview
+
+```{contents} Contents:
+:local: 
+:depth: 3
+```
+
+## Introduction
+
+This guide describes the testing process used by QA at IQSS and provides a reference for others filling in for that role. Please note that many variations are possible, and the main thing is to catch bugs and provide a good quality product to the user community.
+
+## Workflow
+
+Here is a brief description of our workflow: 
+
+### Issue Submission and Prioritization: 
+- Members of the community or the development team submit bugs or request features through GitHub as [Issues](https://github.com/IQSS/dataverse/issues).
+- These Issues are prioritized and added to a two-week-long sprint that can be tracked on the {ref}`kanban-board`.
+
+### Development Process:
+- Developers will work on a solution on a separate branch
+- Once a developer completes their work, they submit a [Pull Request](https://github.com/IQSS/dataverse/pulls) (PR).
+- The PR is reviewed by a developer from the team.
+- During the review, the reviewer may suggest coding or documentation changes to the original developer.
+
+### Quality Assurance (QA) Testing:
+- The QA tester performs a smoke test of core functionality and regression testing.
+- Documentation is used to understand the feature and validate any assertions made.
+- If no documentation is provided in the PR, the tester may refer to the original bug report to determine the desired outcome of the changes.
+- Once the branch is assumed to be safe, it is merged into the develop branch.
+
+### Final Steps:
+- The PR and the Issue are closed and assigned the “merged” status.
+- It is good practice to delete the branch if it is local.
+- The content from the PR becomes part of the codebase for {doc}`future releases </developers/making-releases>`.
+
+The complete suggested workflow can be found at {doc}`qa-workflow`.
+
+## Tips and Tricks
+
+- Start testing simply, with the most obvious test. You don’t need to know all your tests upfront. As you gain comfort and understanding of how it works, try more tests until you are done. If it is a complex feature, jot down your tests in an outline format, some beforehand as a guide, and some after as things occur to you. Save the doc in a testing folder (on Google Drive). This potentially will help with future testing.
+- When in doubt, ask someone. If you are confused about how something is working, it may be something you have missed, or it could be a documentation issue, or it could be a bug! Talk to the code reviewer and the contributor/developer for their opinion and advice.
+- Always tail the server.log file while testing. Open a terminal window to the test instance and `tail -F server.log`. This helps you get a real-time sense of what the server is doing when you interact with the application and makes it easier to identify any stack trace on failure.
+- When overloaded, QA the simple pull requests first to reduce the queue. It gives you a mental boost to complete something and reduces the perception of the amount of work still to be done.
+- When testing a bug fix, try reproducing the bug on the demo server before testing the fix. That way you know you are taking the correct steps to verify that the fix worked.
+- When testing an optional feature that requires configuration, do a smoke test without the feature configured and then with it configured. That way you know that folks using the standard config are unaffected by the option if they choose not to configure it.
+- Back up your DB before applying an irreversible DB update when you are using a persistent/reusable platform. Just in case it fails, and you need to carry on testing something else you can use the backup.
+
+## Release Cadence and Sprints
+
+A release likely spans multiple two-week sprints. Each sprint represents the priorities for that time and is sized so that the team can reasonably complete most of the work on time. This is a goal to help with planning, it is not a strict requirement. Some issues from the previous sprint may remain and likely be included in the next sprint but occasionally may be deprioritized and deferred to another time.
+
+The decision to make a release can be based on the time since the last release, some important feature needed by the community or contractual deadline, or some other logical reason to package the work completed into a named release and posted to the releases section on GitHub.
+
+## Test API
+
+The API test suite is added to and maintained by development. (See {doc}`/developers/testing` in the Developer Guide.) It is generally advisable for code contributors to add API tests when adding new functionality. The approach here is one of code coverage: exercise as much of the code base's code paths as possible, every time to catch bugs. 
+
+This type of approach is often used to give contributing developers confidence that their code didn’t introduce any obvious, major issues and is run on each commit. Since it is a broad set of tests, it is not clear whether any specific, conceivable test is run but it does add a lot of confidence that the code base is functioning due to its reach and consistency. (See {doc}`/qa/test-automation` in the Developer Guide.)
+
+## Making a Release
+
+See {doc}`/developers/making-releases` in the Developer Guide.

doc/sphinx-guides/source/qa/performance-tests.md

@@ -0,0 +1,31 @@
+# Performance Testing
+
+```{contents} Contents:
+:local: 
+:depth: 3
+```
+
+## Introduction
+
+The final testing activity before producing a release is performance testing. This could be done throughout the release cycle but since it is time-consuming, it is done once near the end. Using a load-generating tool named {ref}`Locust <locust>`, our scripts load the statistically most-loaded pages (according to Google Analytics): 50% homepage and 50% some type of dataset page. 
+
+Since dataset page weight also varies by the number of files, a selection of about 10 datasets with varying file counts is used. The pages are called randomly as a guest user with increasing levels of user load, from 1 user to 250 users. Typical daily loads in production are around the 50-user level. Though the simulated user level does have a modest amount of random think time before repeated calls, from 5-20 seconds, it is not a real-world load so direct comparisons to production are not reliable. Instead, we compare performance to prior versions of the product, and based on how that performed in production we have some idea whether this might be similar in performance or whether there is some undetected issue that appears under load, such as inefficient or too many DB queries per page.
+
+## Testing Environment
+
+To run performance tests, we have a performance test cluster on AWS that employs web, database, and Solr. The database contains a copy of production that is updated weekly on Sundays. To ensure the homepage content is consistent between test runs across releases, two scripts set the datasets that will appear on the homepage. There is a script on the web server in the default CentOS user dir and one on the database server in the default CentOS user dir. Run these scripts before conducting the tests. 
+
+Once the performance has been tested and recorded in a [Google spreadsheet](https://docs.google.com/spreadsheets/d/1lwPlifvgu3-X_6xLwq6Zr6sCOervr1mV_InHIWjh5KA/edit?usp=sharing) for this proposed version, the release will be prepared and posted.
+
+## Access
+
+Access to performance cluster instances requires ssh keys. The cluster itself is normally not running to reduce costs. To turn on the cluster, log on to the demo server and run the perfenv scripts from the centos default user dir.
+
+## Special Notes ⚠️ 
+
+Please note the performance database is also used occasionally by members of the Curation team to generate prod reports so a courtesy check with them would be good before taking over the env.
+
+
+Executing the Performance Script
+--------------------------------
+To execute the performance test script, you need to install a local copy of the database-helper-scripts project at <https://github.com/IQSS/dataverse-helper-scripts>. We have since produced a stripped-down script that calls just the collection and dataset pages and works with Python 3.

doc/sphinx-guides/source/qa/qa-workflow.md

@@ -0,0 +1,100 @@
+# QA Workflow for Pull Requests
+
+```{contents} Contents:
+:local: 
+:depth: 3
+```
+## Checklist
+
+1. Assign the PR you are working on to yourself.
+
+1. What does it do?
+
+    Read the description at the top of the PR, any release notes, documentation, and the original issue.
+
+1. Does it address the issue it closes? 
+
+    The PR should address the issue entirely unless otherwise noted.
+
+1. How do you test it?
+
+    Look at the “how to test" section at the top of the pull request. Does it make sense? This likely won’t be the only testing you perform. You can develop further tests from the original issue or problem description, from the description of functionality, the documentation, configuration, and release notes. Also consider trying to reveal bugs by trying to break it: try bad or missing data, very large values or volume of data, exceed any place that may have a limit or boundary.
+
+1. Does it have or need documentation?
+
+    Small changes or fixes usually don’t have docs but new features or extensions of a feature or new configuration options should have documentation.
+
+1. Does it have or need a release note snippet?
+
+    Same as for doc, just a heads up to an admin for something of note or especially upgrade instructions as needed. See also {ref}`writing-release-note-snippets` for what to expect in a release note snippet.
+
+1. Does it include a database migration script (Flyway)?
+
+    First, check the numbering in the filename of the script. It must be in line with the rules defined at {ref}`create-sql-script`. If the number is out of date (very common for older pull requests), do not merge and ask the developer to rename the script. Otherwise, deployment will fail.
+
+    Once you're sure the numbering is ok (the next available number, basically), back up your database and proceeed with testing.
+
+1. Validate the documentation.
+
+    Build the doc using Jenkins or read the automated Read the Docs preview. Does it build without errors?
+    Read it through for sense.
+    Use it for test cases and to understand the feature.
+
+1. Build and deploy the pull request.
+
+    Normally this is done using Jenkins and automatically deployed to the QA test machine. See {ref}`deploy-to-internal`.
+
+1. Configure if required
+
+    If needed to operate and everyone installing or upgrading will use this, configure now as all testing will use it.
+
+1. Smoke test the branch.
+
+    Standard, minimal test of core functionality.
+
+1. Regression test-related or potentially affected features
+
+    If config is optional and testing without config turned on, do some spot checks/ regression tests of related or potentially affected areas. 
+
+1. Configure if optional
+
+    What is the default, enabled or disabled? Is that clearly indicated? Test both.
+    By config here we mean enabling the functionality versus choosing a particular config option. Some complex features have config options in addition to enabling. Those will also need to be tested.
+
+1. Test all the new or changed functionality.
+
+    The heart of the PR, what is this PR adding or fixing? Is it all there and working?
+
+1. Regression test related or potentially affected features.
+
+    Sometimes new stuff modifies and extends other functionality or functionality that is shared with other aspects of the system, e.g. export, import. Check the underlying functionality that was also modified but in a spot check or briefer manner.
+
+1. Report any issues found within the PR
+
+    It can be easy to lose track of what you’ve found, steps to reproduce, and any errors or stack traces from the server log. Add these in a numbered list to a comment in the pr. Easier to check off when fixed and to work on. Add large amounts of text as in the server log as attached, meaningfully named files.
+
+1. Retest all fixes, spot check feature functionality, smoke test
+
+    Similar to your initial testing, it is only narrower.
+
+1. Test upgrade instructions, if required
+
+    Some features build upon the existing architecture but require modifications, such as adding a new column to the DB or changing or adding data. It is crucial that this works properly for our 100+ installations. This testing should be performed at the least on the prior version with basic data objects (collection, dataset, files) and any other data that will be updated by this feature. Using the sample data from the prior version would be good or deploying to dataverse-internal and upgrading there would be a good test. Remember to back up your DB before doing a transformative upgrade so that you can repeat it later if you find a bug.
+
+1. Make sure the API tests in the PR have been completed and passed.
+
+    They are run with each commit to the PR and take approximately 42 minutes to run.
+
+1. Merge PR
+
+    Click the "Merge pull request" button and be sure to use the "Create a merge commit" option to include this PR into the common develop branch.
+
+    Some of the reasons why we encourage using this option over Rebase or Squash are:
+
+    - Preservation of commit history
+    - Clearer context and treaceability
+    - Easier collaboration, bug tracking and reverting
+
+1. Delete merged branch
+
+    Just a housekeeping move if the PR is from IQSS. Click the delete branch button where the merge button had been. There is no deletion for outside contributions.