[Need QA Prep] Feature/The Node-Preprint Divorce [PLAT-555][PLAT-480][PLAT-559][PLAT-561][PLAT-1168]

[pattisdr]

Reticketed from #8422

Related PR's

Back End
Ember-OSF
Front End
TODO List

❗️ After running data migration, you need to run this elastic search migration script

Purpose

Removes the dependency of Preprints on Nodes. Nodes can house supplemental files for a preprint, but beyond that, Preprints are completely independent.

Ticket contains:

• Backend changes for node-preprint divorce
• Schema and data migrations for legacy preprints
• v1/Non-Ember front-end changes
• API changes for preprints
• Docs here Support pages will also need to be updated (Becca?).

How to create a preprint in the API

Step 1: Start Preprint

POST http://localhost:8000/v2/preprints/
Required fields are title and provider

request body
 {
     "data": {
       "type": "preprints",
       "attributes": {
        "title": "My first preprint",
        "tags": ["coffee", "cinnamon"]
       },
       "relationships": {
          "provider": {
             "data": {
                "type": "providers",
                "id": "osf"
             
             }
          }
    		
       }
     }
}

Step 2 Upload new file to preprint, or copy file from existing node:

Uploading new file

GET http://localhost:8000/v2/preprints/nmqp6/files/ and then

curl -X "PUT" "http://localhost:7777/v1/resources/nmqp6/providers/osfstorage/?kind=file&name=my_preprint_file.txt" \
--
-H "Authorization: Bearer my-token-goes-here" \
-H "Content-Type: text/plain" \
-d "This is a text file"

Copying file from supplementary node

Get "move" link from supplementary node and then POST to move link where resource is the preprint id, action is copy, and provider is osfstorage.
GET http://localhost:8000/v2/nodes/4u3mk/files/osfstorage/ and then

curl -X "POST" "http://localhost:7777/v1/resources/4u3mk/providers/osfstorage/5b2803b04c9d8502e5aea0cf" \
--
-H "Authorization: Bearer my-token-goes-here " \
-H "Content-Type: application/json" \
-d '{"action": "copy", "path": "/", "resource": "nmqp6", "provider": "osfstorage", "conflict": "replace"}'

Step 3 - Update preprint to set newly uploaded file as primary file, add subjects if not there already, (supplementary node optional) and publish

PATCH http://localhost:8000/v2/preprints/nmqp6/
 {
     "data": {
       "id": "nmqp6",
       "type": "preprints",
       "attributes": {
        "subjects": [["5b2419881f4ae00001670157"]],
        "is_published": "true"
       },
       "relationships": {
          "primary_file": {
             "data": {
                "type": "files",
                "id": "5b2809294c9d8502e5aea0e1"
             }
          },
          "node": {
             "data": {
                 "type": "nodes",
                 "id": "4u3mk"
             }
          }
    		
       }
     }
}

Changes

Too many to list, here are some of them.

Admin App

• Preprints are their own thing in the admin app, not relying on node fields
• You can modify contributors directly, flag preprints as spam/ham, and reindex preprints in elastic search and SHARE
• Several node templates/views made more generic to work with nearly-identical preprints, so double check deleting nodes, marking as spam/ham, reindexing in elasticsearch/SHARE

API

• Preprint endpoints are looking at preprint fields, not node fields.
• PreprintContributors/PreprintDetail endpoints pulling from PreprintContributors
• PreprintStorageList endpoint and PreprintFiles list endpoint added (to expose OSFStorage Waterbutler endpoints for uploading primary file. PreprintStorageList just shows OSFStorage, and PreprintFiles endpoint show all files on the preprint. If going through UI, should just have primary file, but if uploading to wb directly, is possible to see more files.
• New process for creating a preprint 1) POST to v2/preprints/ (title, provider are required fields) 2) Use WB links to upload or move file to created preprint 3) PATCH to v2/preprints/<preprint_id>/ to finish up process.
• Preprint node relationship endpoint added for removing node from preprint
• Original publication date can now be removed from the preprint.
• New preprints scopes added now that preprints/nodes are separate

Elastic Search

• Legacy nodes attached to preprints are reindexed as nodes in a one-off script (scripts/remove_after_use/node_preprint_es.py)
• Legacy preprints are added to the index as preprints
• Moving forward, nodes are indexed as nodes, preprints are indexed as preprints
• Modifications to search migration to add indexing for preprints and preprint files, and to stop indexing nodes as preprints

Modeling

• New PreprintLog and PreprintContributor models
• PreprintService model is renamed to Preprint Model
• New ContributorMixin to share contributor-related methods between Nodes and Preprints - Node guardian/groups work (future feature), further consolidates this code.
• New SpamOverrideMixin to share spam-methods between Nodes and Preprints
• Preprints uses Django-guardian for contributor permissions
• Write contributors have been upgraded to have more permissions - can edit most things on a submitted preprint except publishing, withdrawing, etc.
• Preprints do not have addons, so a few OSFStorage methods have been moved to the preprint.
• UploadMixin added to Preprint that creates a root folder for storage

Preprint File

• Stored on preprint instead of node
• Some OSFStorage methods modified to handle permission checking on preprint in addition to node
• Preprints have regions (by default, same region as UserSettings, in migration, pulls region off of node).

Legacy UI

• Dashboard My Preprints fetches from users/preprints/ instead of users/nodes/?filter['preprint']=True. Activity logs removed
• Preprint file restrictions removed from fangorn
• Language changed on project page to reflect that project is a supplemental file for a project

Supplemental project attached to withdrawn preprint - status says Withdrawn

Withdrawn preprint still shows up for a user on My Projects page. Is marked as withdrawn.

Published preprints show up on My Projects page

My private preprints still show up on My Projects page

Selecting multiple preprints on My Projects page

Elastic search before migration - 2 preprints 3 projects

Elastic search after migration - 2 preprints 5 projects (2 projects formerly indexed as preprints were reindexed as projects)

Preprint files don't have links, but you can click on preprint link. File detail for preprint page is blocked.

QA Notes

Many many things to test. A lot of Node code was adapted to work for preprints so we want to make sure we didn't break things. Preprints front-end is not addressed here.

To check on Node

• Regression of search results - OSF side, for preprints, preprint files
• That file logs still work, (osfstorage_file_added, osfstorage_file_copied, for example). Do all the file operations on a node, verify logs are what you expect.
• Contributor operations on projects still work.
• Project page banners that your node is a supplementary file for a preprint are what you expect, and appear when you expect. Test for various providers.
• Verify that you get proper warnings when trying to remove a project/component that is a supplemental project for a preprint, or when trying to make it private
• MyProjects page shows what you would expect for preprints. Activity log has been removed since we're not exposing preprint logs yet.
• Searching for nodes still works the way it used to
• SHARE updates for projects still work the way you would expect.
• Test nodes in admin app, removing contributors, marking node as deleted, marking nodes as spam/ham,
• Make sure deleting nodes with children on them still works
• Test node contributor emails

To check on preprint

• Preprints themselves, instead of nodes are indexed in elastic search
• Preprints are indexed in SHARE correctly (HAVE NOT TESTED THIS LOCALLY)
• Test preprints in admin app, removing contributors, marking preprint as deleted, marking preprints as spam/ham, reindexing preprint in elastic search
• Preprint files only show up in search if preprint is verified publishable
• If you change the preprint file, the original is removed from search
• Preprint files don't have links to file in search
• Full regression on preprint dois
• Test preprint contributor emails (text makes sense, not directing users to project page to change contributors, for example).

Checking migration

• Make several preprints ahead of time, make sure they have a title, abstract, primary_file, multiple contributors with different permissions/bibligraphic, article_doi, some public, others private, some that have been deleted, some that have been abandoned, tags, some whose nodes have been marked as spam, modified date.
• modified date of preprint will be modified to the date of a preprint-related node log if it is a later date than the preprint modified.
• Make sure that all of these files are moved over to the preprint.

Side Effects

• Preprint titles were increased to 512 chars max limit. Node title limits were similarly increased. If there's a reason nodes still need to be 200, I can do this.

Ticket

[caseyrollins]

Latest commits headed to staging3

[pattisdr]

Can you get these latest few commits on staging3 @sloria ?

[sloria]

Latest commits are headed to staging3.

[pattisdr]

ah, thanks @sloria, I broke this when removing the AbstractNode import so this would still work for preprints and missed the line below. I couldn't duplicate the Recent Activity issue because I never looked at this for components. f4ecdb8

[pattisdr]

Allows logged-in users to claim an unregistered account as their own (following link from a preprints email)

[pattisdr]

must_be_valid_project was modified as a part of the divorce if preprints_valid=True passed in, then it would work for preprints. called _inject_nodes will add kwargs['node'] = Preprint object. Then claim_user_registered will have the Preprint object as node, and all methods in that function will work for a node or preprint.

There might be some confusion over "node" language referring to a preprint - at some point, we might want to generalize the language in a few places that nodes and preprints both use, (resource, or similar), but I think this is low priority.

[sloria]

This is merged into master via feature/node-preprint