Locally FAIR Data

doc/release-notes/12319-LocallyFAIRdata

@@ -0,0 +1,9 @@
+This release includes experimental support for "Locally FAIR" data.
+This feature allows publication of content that will only be visible to authorized users or groups within a Dataverse installation.
+User without authorization will not see the Locally FAIR collections, datasets, or files in search results and cannot visit their
+pages or access them via the Dataverse API.
+
+For more information, see the [Locally FAIR Data](https://guides.dataverse.org/en/latest/user/locally-fair-data.html) guide.
+
+New Config Option:
+Whether Locally FAIR content can be created is controlled by the new `dataverse.feature.allow-locally-fair-data` feature flag.

doc/sphinx-guides/source/api/native-api.rst

@@ -929,6 +929,116 @@ In particular, the user permissions that this API call checks, returned as boole
 
   curl -H "X-Dataverse-key: $API_TOKEN" -X GET "$SERVER_URL/api/dataverses/$ID/userPermissions"
 
+  List Locally FAIR Role Assignees for a Dataverse Collection
+  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+  Lists the Locally FAIR role assignee identifiers configured for a Dataverse collection identified by ``id``.
+  For more about the concept, see the :doc:`/user/locally-fair` section of the User Guide.
+
+  This API is superuser-only.
+
+  .. code-block:: bash
+
+    export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
+    export SERVER_URL=https://demo.dataverse.org
+    export ID=root
+
+    curl -H "X-Dataverse-key:$API_TOKEN" "$SERVER_URL/api/dataverses/$ID/locallyFairRoleAssignees"
+
+  The fully expanded example above (without environment variables) looks like this:
+
+  .. code-block:: bash
+
+    curl -H "X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" "https://demo.dataverse.org/api/dataverses/root/locallyFairRoleAssignees"
+
+  The response is a JSON array of role assignee identifiers. For example:
+
+  .. code-block:: json
+
+    [
+      "@TestUser",
+      "&maildomain/harvard.edu"
+    ]
+
+  Set Locally FAIR Role Assignees for a Dataverse Collection
+  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+  Replaces the full set of locally FAIR role assignee identifiers for a Dataverse collection identified by ``id``.
+
+  This API is superuser-only.
+
+  .. code-block:: bash
+
+    export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
+    export SERVER_URL=https://demo.dataverse.org
+    export ID=root
+    export JSON='["@TestUser","&maildomain/harvard.edu"]'
+
+    curl -H "X-Dataverse-key:$API_TOKEN" -X PUT -H "Content-Type: application/json" "$SERVER_URL/api/dataverses/$ID/locallyFairRoleAssignees" -d "$JSON"
+
+  The fully expanded example above (without environment variables) looks like this:
+
+  .. code-block:: bash
+
+    curl -H "X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" -X PUT -H "Content-Type: application/json" "https://demo.dataverse.org/api/dataverses/root/locallyFairRoleAssignees" -d '["@TestUser","&maildomain/harvard.edu"]'
+
+  Pass an empty array to clear all locally FAIR role assignees from the collection:
+
+  .. code-block:: bash
+
+    curl -H "X-Dataverse-key:$API_TOKEN" -X PUT -H "Content-Type: application/json" "$SERVER_URL/api/dataverses/$ID/locallyFairRoleAssignees" -d '[]'
+
+  All supplied identifiers must be valid existing role assignee identifiers. Invalid identifiers will result in ``400 Bad Request``.
+
+  Add a Locally FAIR Role Assignee to a Dataverse Collection
+  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+  Adds a single locally FAIR role assignee identifier to a Dataverse collection identified by ``id``.
+
+  This API is superuser-only.
+
+  .. code-block:: bash
+
+    export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
+    export SERVER_URL=https://demo.dataverse.org
+    export ID=root
+    export ROLE_ASSIGNEE=&shib/1
+
+    curl -H "X-Dataverse-key:$API_TOKEN" -X PUT "$SERVER_URL/api/dataverses/$ID/locallyFairRoleAssignees/$ROLE_ASSIGNEE"
+
+  The fully expanded example above (without environment variables) looks like this:
+
+  .. code-block:: bash
+
+    curl -H "X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" -X PUT "https://demo.dataverse.org/api/dataverses/root/locallyFairRoleAssignees/&shib/1"
+
+  The response includes the updated set of locally FAIR role assignee identifiers.
+
+  Delete a Locally FAIR Role Assignee from a Dataverse Collection
+  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+  Removes a single locally FAIR role assignee identifier from a Dataverse collection identified by ``id``.
+
+  This API is superuser-only.
+
+  .. code-block:: bash
+
+    export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
+    export SERVER_URL=https://demo.dataverse.org
+    export ID=root
+    export ROLE_ASSIGNEE=:authenticated-users
+
+    curl -H "X-Dataverse-key:$API_TOKEN" -X DELETE "$SERVER_URL/api/dataverses/$ID/locallyFairRoleAssignees/$ROLE_ASSIGNEE"
+
+  The fully expanded example above (without environment variables) looks like this:
+
+  .. code-block:: bash
+
+    curl -H "X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" -X DELETE "https://demo.dataverse.org/api/dataverses/root/locallyFairRoleAssignees/:authenticated-users"
+
+  The response includes the updated set of locally FAIR role assignee identifiers. Removing an identifier that is blank or not currently assigned will result in ``400 Bad Request``.
+
+
 .. _create-dataset-command:
 
 Create a Dataset in a Dataverse Collection

doc/sphinx-guides/source/installation/config.rst

@@ -4082,6 +4082,15 @@ dataverse.feature.require-embargo-reason
 
 Require an embargo reason when a user creates an embargo on one or more files. See :ref:`embargoes`.
 
+.. _dataverse.feature.allow-locally-fair-data:
+
+dataverse.feature.allow-locally-fair-data
++++++++++++++++++++++++++++++++++++++++++
+
+Allows support for Locally FAIR collections and datasets.
+When enabled, selected content can remain visible only to authorized users or groups within a Dataverse installation.
+See :doc:`/user/locally-fair` for more information.
+
 .. _:ApplicationServerSettings:
 
 Application Server Settings

doc/sphinx-guides/source/user/index.rst

@@ -21,6 +21,7 @@ User Guide
    dataverse-management
    dataset-management
    tabulardataingest/index
+   locally-fair
    appendix
 
 .. |what-is-dataverse| image:: ./img/what-is-dataverse.svg

doc/sphinx-guides/source/user/locally-fair.rst

@@ -0,0 +1,106 @@
+Locally FAIR
+++++++++++++
+
+Locally FAIR describes content that is managed according to FAIR principles
+(Findable, Accessible, Interoperable, and Reusable) within a defined local or
+organizational community rather than for the public internet as a whole.
+
+Dataverse now has optional, experimental support for managing Locally FAIR collections.
+
+In a typical public Dataverse installation, published dataset metadata is visible
+to everyone, even if the dataset's files themselves may be embargoed or restricted. Locally FAIR support
+extends this model by allowing some collections, and the published datasets within them to remain
+visible only to designated users or groups. This makes it possible for a single
+Dataverse installation to support both:
+
+- public, globally discoverable content; and
+- organizational content whose existence and metadata are only be visible to
+  authorized users.
+
+The rationale for making some content Locally FAIR can vary.
+Locally FAIR content can include:
+
+- sensitive research collections;
+- institution-only datasets;
+- datasets that should not be accessible to bots that may not adhere to the dataset license and terms, and
+- projects under contractual or policy restrictions;
+
+Dataverse's Locally FAIR mechanism is appropriate for repositories that will house at least some data
+whose metadata should only be visible to organizational members. The decision to make data Locally FAIR
+is managed at the collection level and repositories can have both FAIR and Locally FAIR content.
+
+.. contents:: |toctitle|
+   :local:
+
+What Locally FAIR Means
+=======================
+
+Locally FAIR content is intended to be FAIR within a particular community.
+
+That means:
+- **Findable** Data is easy to locate for both humans and machines, when authorized. Locally FAIR datasets (and files if configured) have persistent identifiers, but do not use DOIs which are publicly searchable.
+
+- **Accessible** Data is retrievable through standardized protocols. Authorized users can use Dataverse's standard user interface and API calls to access Locally FAIR content in the same way they do with any published data.
+
+- **Interoperable** Data should be compatible with other datasets and systems. Locally FAIR datasets in Dataverse use the same standard metadata blocks as for public content and files undergo the same ingest process, use the same previewers and tools, etc.
+
+- **Reusable** Data should be well-described and licensed in a way that allows others to use it for future research. The licenses and terms on locally FAIR content make it clear how and when the data can be re-used.
+
+Why Repositories Use It
+=======================
+
+Without Locally FAIR support, repositories may need separate Dataverse
+installations to separate public and organization-only content.
+
+How It Differs from Restricted Files
+====================================
+
+Restricting or embargoing files limits access to the file contents, but in a standard public
+repository the dataset's published metadata, including the list of files, would still be visible.
+If a dataset allows requests for file access, anyone can request access, even if the dataset's
+license or terms limit access to specific groups.
+
+Locally FAIR goes further. Locally FAIR collections and datasets do not appear in content listings or
+search results for unauthorized users, nor can the collection/dataset/file page be viewed. API access
+is also blocked for unauthorized access.
+
+Who Can See Locally FAIR Content
+================================
+
+Visibility is determined by superusers and is managed at the collection level.
+Access can be granted to any group(s) or user(s) defined in Dataverse - the same groups/users
+available when assigning roles on collections, datasets, and files.
+
+How Can You Tell When Content is Locally FAIR?
+==============================================
+
+The Dataverse UI adds a "Locally FAIR" tag to all collections, datasets, and files who's visibility
+is limited by the locally FAIR mechanism.
+
+
+Why is Locally FAIR support "Experimental"
+==========================================
+
+The word "experimental" is used when functionality is new, may evolve signifcantly in future releases,
+and generally may require more effort to configure and manage and/or more effort to support than more
+mature functionality.
+
+With the current Locally FAIR implementation, managers need to be aware that they are responsible for
+choosing collection settings compatible with Locally FAIR content, i.e. not using DOIs (whose metadata
+is publicly accessible) or publicly visible stores, etc. Users and managers should also be aware that
+some functionality that might expose Locally FAIR content, e.g. linking, may not be prohibited programmatically
+but should still be avoided. Similarly, users should be aware that functionality such as metrics and quotas
+may expose the existence of Locally FAIR content. If your Dataverse instance supports Locally FAIR data,
+you are encouraged to be an active participant in reporting any issues and suggesting further improvements.
+
+Things to Keep in Mind
+======================
+
+If your repository supports Locally FAIR content:
+
+- published does not always mean public;
+- search and browse results may vary depending on who is logged in;
+- colleagues outside your authorized group may not be able to see the same
+  datasets you can see;
+- you should not share Locally FAIR content with others who don't have access themselves; and
+- this functionality is experimental.

src/main/java/edu/harvard/iq/dataverse/DatasetPage.java

@@ -2139,9 +2139,15 @@ private String init(boolean initFull) {
                 return permissionsWrapper.notFound();
             }
 
-            // Check permisisons
-            if (!(workingVersion.isReleased() || workingVersion.isDeaccessioned()) && !this.canViewUnpublishedDataset()) {
-                return permissionsWrapper.notAuthorized();
+            // Check permissions
+            boolean releasedAndCanView = workingVersion.isReleased() && (!dataset.isLocallyFAIR() || permissionsWrapper
+                    .hasLocallyFAIRAccess(dvRequestService.getDataverseRequest(), dataset));
+            if (!(releasedAndCanView || workingVersion.isDeaccessioned()) && !this.canViewUnpublishedDataset()) {
+                if (dataset.isLocallyFAIR()) {
+                    return permissionsWrapper.notFound();
+                } else {
+                    return permissionsWrapper.notAuthorized();
+                }
             }
 
             if (retrieveDatasetVersionResponse != null && !retrieveDatasetVersionResponse.wasRequestedVersionRetrieved()) {

src/main/java/edu/harvard/iq/dataverse/Dataverse.java

@@ -16,7 +16,9 @@
 import java.util.Objects;
 import java.util.Set;
 import jakarta.persistence.CascadeType;
+import jakarta.persistence.CollectionTable;
 import jakarta.persistence.Column;
+import jakarta.persistence.ElementCollection;
 import jakarta.persistence.Entity;
 import jakarta.persistence.EnumType;
 import jakarta.persistence.Enumerated;
@@ -105,7 +107,40 @@ public enum DataverseType {
     @NotNull(message = "{dataverse.category}")
     @Column( nullable = false )
     private DataverseType dataverseType;
-
+
+
+    @ElementCollection
+    @CollectionTable(name = "dataverse_locallyfairassignees",
+        joinColumns = @JoinColumn(name = "dataverse_id"))
+    @Column(name = "assigneeidentifier")
+    private Set<String> locallyFAIRRoleAssigneeIdentifiers = new HashSet<>();
+
+    @Override
+    public Set<String> getLocallyFAIRRoleAssigneeIdentifiers() {
+        return locallyFAIRRoleAssigneeIdentifiers;
+    }
+
+    public void setLocallyFAIRRoleAssigneeIdentifiers(Set<String> roleAssigneeIdentifiers) {
+        this.locallyFAIRRoleAssigneeIdentifiers = roleAssigneeIdentifiers;
+    }
+
+    public void addLocallyFAIRRoleAssignee(String assigneeIdentifier) {
+        if (locallyFAIRRoleAssigneeIdentifiers == null) {
+            locallyFAIRRoleAssigneeIdentifiers = new HashSet<>();
+        }
+        locallyFAIRRoleAssigneeIdentifiers.add(assigneeIdentifier);
+    }
+
+    public void removeLocallyFAIRRoleAssignee(String assigneeIdentifier) {
+        if (locallyFAIRRoleAssigneeIdentifiers != null) {
+            locallyFAIRRoleAssigneeIdentifiers.remove(assigneeIdentifier);
+        }
+    }
+
+    public boolean LocallyFAIR(String assigneeIdentifier) {
+        return locallyFAIRRoleAssigneeIdentifiers != null && locallyFAIRRoleAssigneeIdentifiers.contains(assigneeIdentifier);
+    }
+
     /**
      * When {@code true}, users are not granted permissions the got for parent
      * dataverses.
@@ -907,7 +942,7 @@ public boolean isAncestorOf( DvObject other ) {
         }
         return false;
     }
-    
+
     public String getLocalURL() {
         return  SystemConfig.getDataverseSiteUrlStatic() + "/dataverse/" + this.getAlias();
     }
@@ -924,4 +959,10 @@ public void addInputLevelsMetadataBlocksIfNotPresent(List<DataverseFieldTypeInpu
     private boolean hasMetadataBlock(MetadataBlock metadataBlock) {
         return metadataBlocks.stream().anyMatch(block -> block.getId().equals(metadataBlock.getId()));
     }
+
+    @Override
+    public boolean isLocallyFAIR() {
+        return !locallyFAIRRoleAssigneeIdentifiers.isEmpty();
+    }
+
 }