Merge branch 'develop' into file-metadata-provenance-2295

doc/sphinx-guides/source/_static/installation/files/etc/selinux/targeted/src/policy/domains/misc/shibboleth.te

@@ -0,0 +1,15 @@
+module shibboleth 1.0;
+
+require {
+    class file {open read};
+    class sock_file write;
+    class unix_stream_socket connectto;
+    type httpd_t;
+    type initrc_t;
+    type var_run_t;
+    type var_t;
+}
+
+allow httpd_t initrc_t:unix_stream_socket connectto;
+allow httpd_t var_run_t:sock_file write;
+allow httpd_t var_t:file {open read};

doc/sphinx-guides/source/developers/index.rst

@@ -20,3 +20,4 @@ Contents:
    making-releases
    tools
    unf/index
+   selinux

doc/sphinx-guides/source/developers/selinux.rst

@@ -0,0 +1,110 @@
+=======
+SELinux
+=======
+
+.. contents:: :local:
+
+Introduction
+------------
+
+The ``shibboleth.te`` file below that is mentioned in the :doc:`/installation/shibboleth` section of the Installation Guide was created on CentOS 6 as part of https://github.com/IQSS/dataverse/issues/3406 but may need to be revised for future versions of RHEL/CentOS. The file is versioned with the docs and can be found in the following location:
+
+``doc/sphinx-guides/source/_static/installation/files/etc/selinux/targeted/src/policy/domains/misc/shibboleth.te``
+
+.. literalinclude:: ../_static/installation/files/etc/selinux/targeted/src/policy/domains/misc/shibboleth.te
+   :language: text
+
+This document is something of a survival guide for anyone who is tasked with updating this file.
+
+Development Environment
+-----------------------
+
+In order to work on the ``shibboleth.te`` file you need to ``ssh`` into a RHEL or CentOS box running Shibboleth (instructions are in the :doc:`/installation/shibboleth` section of the Installation Guide) such as https://beta.dataverse.org or https://demo.dataverse.org that has all the commands below installed. As of this writing, the ``policycoreutils-python`` RPM was required.
+
+Recreating the shibboleth.te File
+---------------------------------
+
+If you're reading this page because someone has reported that Shibboleth doesn't work with SELinux anymore (due to an operating system upgrade, perhaps) you *could* start with the existing ``shibboleth.te`` file, but it is recommended that you create a new one instead to ensure that extra lines aren't included that are no longer necessary.
+
+The file you're recreating is called a Type Enforcement (TE) file, and you can read more about it at https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Security-Enhanced_Linux/chap-Security-Enhanced_Linux-SELinux_Contexts.html
+
+The following doc may or may not be helpful to orient you: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Security-Enhanced_Linux/sect-Security-Enhanced_Linux-Fixing_Problems-Allowing_Access_audit2allow.html
+
+Ensure that SELinux is Enforcing
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If ``getenforce`` returns anything other than ``Enforcing``, run ``setenforce Enforcing`` or otherwise configure SELinux by editing ``/etc/selinux/config`` and rebooting until SELinux is enforcing.
+
+Removing the Existing shibboleth.te Rules
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Use ``semodule -l | grep shibboleth`` to see if the ``shibboleth.te`` rules are already installed. Run ``semodule -r shibboleth`` to remove the module, if necessary. Now we're at square one (no custom rules) and ready to generate a new ``shibboleth.te`` file.
+
+Exercising SELinux denials
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+As of this writing, there are two optional components of Dataverse that are known not to work with SELinux out of the box with SELinux: Shibboleth and rApache.
+
+We will be exercising SELinux denials with Shibboleth, and the SELinux-related issues are expected out the box:
+
+- Problems with the dropdown of institutions being created on the Login Page ("Internal Error - Failed to download metadata from /Shibboleth.sso/DiscoFeed.").
+- Problems with the return trip after you've logged into HarvardKey or whatever ("shibsp::ListenerException" and "Cannot connect to shibd process, a site adminstrator should be notified.").
+
+In short, all you need to do is try to log in with Shibboleth and you'll see problems associated with SELinux being enabled.
+
+Stub out the new shibboleth.te file
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Iterate on the new ``shibboleth.te`` file wherever you like, such as the root user's home directory in the example below. Start by adding a ``module`` line like this:
+
+``echo 'module shibboleth 1.0;' > /root/shibboleth.te``
+
+Note that a version is required and perhaps it should be changed, but we'll stick with ``1.0`` for now. The point is that the ``shibboleth.te`` file must begin with that "module" line or else the ``checkmodule`` command you'll need to run later will fail. Your file should look like this:
+
+.. code-block:: text
+
+        module shibboleth 1.0;
+        # require lines go here
+        # allow lines go here
+
+Iteratively Use audit2allow to Add Rules and Test Your Change
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Now that ``shibboleth.te`` has been stubbed out, we will iteratively add lines to it from the output of piping SELinux Access Vector Cache (AVC) denial messages to ``audit2allow -r``. These errors are found in ``/var/log/audit/audit.log`` so tail the file as you attempt to log in to Shibboleth.
+
+``# tail -f /var/log/audit/audit.log | fgrep type=AVC``
+
+You should see messages that look something like this:
+
+``type=AVC msg=audit(1476728970.378:271405): avc:  denied  { write } for  pid=28548 comm="httpd" name="shibd.sock" dev=dm-2 ino=393300 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:var_run_t:s0 tclass=sock_file``
+
+Next, pipe these message to ``audit2allow -r`` like this:
+
+``echo 'type=AVC msg=audit(1476728970.378:271405): avc:  denied  { write } for  pid=28548 comm="httpd" name="shibd.sock" dev=dm-2 ino=393300 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:var_run_t:s0 tclass=sock_file' | audit2allow -r``
+
+This will produce output like this:
+
+.. code-block:: text
+
+        require {
+                type var_run_t;
+                type httpd_t;
+                class sock_file write;
+        }
+
+        #============= httpd_t ==============
+        allow httpd_t var_run_t:sock_file write;
+
+Copy and paste this output into the ``shibboleth.te`` file you stubbed out above. Then, use the same ``checkmodule``, ``semodule_package``, and ``semodule`` commands documented in the :doc:`/installation/shibboleth` section of the Installation Guide on your file to activate the SELinux rules you're constructing.
+
+Once your updated SELinux rules are in place, try logging in with Shibboleth again. You should see a different AVC error. Pipe that error into ``audit2allow -r`` as well and put the resulting content into the ``shibboleth.te`` file you're constructing. As you do this, manually reformat the file using the following rules:
+
+- Put the ``require`` block at the top.
+- Within the require block, sort the lines.
+- Put the ``allow`` lines at the bottom and sort them.
+- Where possible, avoid duplicate lines by combining operations such as ``open`` and ``read`` into ``{open read}``.
+- Remove all comment lines.
+
+Keep iterating until it works and then create a pull request based on your updated file. Good luck!
+
+Many thanks to Bill Horka from IQSS for his assistance in explaining how to construct a SELinux Type Enforcement (TE) file!

doc/sphinx-guides/source/installation/r-rapache-tworavens.rst

@@ -19,6 +19,9 @@ Disable SELinux on httpd:
 
 ``getenforce``
 
+(Note: a pull request to get rApache working with SELinux is welcome! Please see the :doc:`/developers/selinux` section of the Developer Guide to get started.)
+
+
 https strongly recommended; signed certificate (as opposed to self-signed) is recommended. 
 
 Directory listing needs to be disabled on the web documents folder served by Apache: 

doc/sphinx-guides/source/installation/shibboleth.rst

@@ -201,10 +201,56 @@ attribute-map.xml
 
 By default, some attributes ``/etc/shibboleth/attribute-map.xml`` are commented out. Edit the file to enable them so that all the require attributes come through. You can download a `sample attribute-map.xml file <../_static/installation/files/etc/shibboleth/attribute-map.xml>`_.
 
+Disable or Reconfigure SELinux
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+SELinux is set to "enforcing" by default on RHEL/CentOS, but unfortunately Shibboleth does not "just work" with SELinux. You have two options. You can disable SELinux or you can reconfigure SELinux to accommodate Shibboleth.
+
 Disable SELinux
-~~~~~~~~~~~~~~~
+^^^^^^^^^^^^^^^
+
+The first and easiest option is to set ``SELINUX=permisive`` in ``/etc/selinux/config`` and run ``setenforce permissive`` or otherwise disable SELinux to get Shibboleth to work. This is apparently what the Shibboleth project expects because their wiki page at https://wiki.shibboleth.net/confluence/display/SHIB2/NativeSPSELinux says, "At the present time, we do not support the SP in conjunction with SELinux, and at minimum we know that communication between the mod_shib and shibd components will fail if it's enabled. Other problems may also occur."
+
+Reconfigure SELinux to Accommodate Shibboleth
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The second (more involved) option is to use the ``checkmodule``, ``semodule_package``, and ``semodule`` tools to apply a local policy to make Shibboleth work with SELinux. Let's get started.
+
+Put Type Enforcement (TE) File in misc directory
+````````````````````````````````````````````````
+
+Copy and paste or download the `shibboleth.te <../_static/installation/files/etc/selinux/targeted/src/policy/domains/misc/shibboleth.te>`_ Type Enforcement (TE) file below and put it at ``/etc/selinux/targeted/src/policy/domains/misc/shibboleth.te``.
+
+.. literalinclude:: ../_static/installation/files/etc/selinux/targeted/src/policy/domains/misc/shibboleth.te
+   :language: text
+
+(If you would like to know where the ``shibboleth.te`` came from and how to hack on it, please see the :doc:`/developers/selinux` section of the Developer Guide. Pull requests are welcome!)
+
+Navigate to misc directory
+``````````````````````````
+
+``cd /etc/selinux/targeted/src/policy/domains/misc``
+
+Run checkmodule
+```````````````
+
+``checkmodule -M -m -o shibboleth.mod shibboleth.te``
+
+Run semodule_package
+````````````````````
+
+``semodule_package -o shibboleth.pp -m shibboleth.mod``
+
+Silent is golden. No output is expected.
+
+Run semodule
+````````````
+
+``semodule -i shibboleth.pp``
+
+Silent is golden. No output is expected. This will place a file in ``/etc/selinux/targeted/modules/active/modules/shibboleth.pp`` and include "shibboleth" in the output of ``semodule -l``. See the ``semodule`` man page if you ever want to remove or disable the module you just added.
 
-You must set ``SELINUX=permisive`` in ``/etc/selinux/config`` and run ``setenforce permissive`` or otherwise disable SELinux for Shibboleth to work. "At the present time, we do not support the SP in conjunction with SELinux, and at minimum we know that communication between the mod_shib and shibd components will fail if it's enabled. Other problems may also occur." -- https://wiki.shibboleth.net/confluence/display/SHIB2/NativeSPSELinux
+Congrats! You've made the creator of http://stopdisablingselinux.com proud. :)
 
 Restart Apache and Shibboleth
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

scripts/migration/migrate_passwords.sql

@@ -0,0 +1,5 @@
+update builtinuser 
+set passwordencryptionversion = 0,
+encryptedpassword= _dvn3_vdcuser.encryptedpassword
+from _dvn3_vdcuser
+where _dvn3_vdcuser.username=builtinuser.username;

scripts/migration/migration_instructions.txt

@@ -110,7 +110,7 @@ datafile_pub_date.sql
 
 12. (when ready for users to log in) add user passwords
 
-[how?]
+migrate_passwords.sql
 
 __________________________________________________