Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IQSS/10318 Uningest/Reingest UI #10319

Merged
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
4 changes: 4 additions & 0 deletions doc/sphinx-guides/source/api/native-api.rst
Original file line number Diff line number Diff line change
Expand Up @@ -2854,6 +2854,8 @@ The fully expanded example above (without environment variables) looks like this

curl -H "X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" -X PUT -d true "https://demo.dataverse.org/api/files/:persistentId/restrict?persistentId=doi:10.5072/FK2/AAA000"

.. _file-uningest:

Uningest a File
~~~~~~~~~~~~~~~

Expand Down Expand Up @@ -2891,6 +2893,8 @@ The fully expanded example above (without environment variables) looks like this

curl -H "X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" -X POST "https://demo.dataverse.org/api/files/:persistentId/uningest?persistentId=doi:10.5072/FK2/AAA000"

.. _file-reingest:

Reingest a File
~~~~~~~~~~~~~~~

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -61,16 +61,16 @@ Uningest and Reingest
=====================

Ingest will only work for files whose content can be interpreted as a table.
Multi-sheets spreadsheets and CSV files with different number of entries per row are two examples where ingest will fail.
Multi-sheet spreadsheets and CSV files with a different number of entries per row are two examples where ingest will fail.
This is non-fatal. The Dataverse software will not produce a .tab version of the file and will show a warning to users
who can see the draft version of the dataset containing the file that will indicate why ingest failed. When the file is published as
part of the dataset, there will be no indication that ingest was attempted and failed.

If the warning message is a concern, the Dataverse software includes both an API call (see the Files section of the :doc:`/api/native-api` guide)
and an Edit/Uningest menu option displayed on the file page, that allow a file to be Uningested. These are only available to superusers.
If the warning message is a concern, the Dataverse software includes both an API call (see :ref:`file-uningest` in the :doc:`/api/native-api` guide)
and an Edit/Uningest menu option displayed on the file page, that allow a file to be uningested. These are only available to superusers.
Uningest will remove the warning. Uningest can also be done for a file that was successfully ingested.
This will remove the .tab version of the file that was generated.

If a file is a tabular format but was never ingested, .e.g. due to the ingest file size limit being lower in the past, or if ingest had failed,
e.g. in a prior Dataverse version, an reingest API (see the Files section of the :doc:`/api/native-api` guide) and a file page Edit/Reingest option
e.g. in a prior Dataverse version, an reingest API (see :ref:`file-reingest` in the :doc:`/api/native-api` guide) and a file page Edit/Reingest option
in the user interface allow ingest to be tried again. As with Uningest, this fucntionality is only available to superusers.
39 changes: 22 additions & 17 deletions src/main/java/edu/harvard/iq/dataverse/FilePage.java
Original file line number Diff line number Diff line change
Expand Up @@ -484,7 +484,7 @@ public String restrictFile(boolean restricted) throws CommandException{
public String ingestFile() throws CommandException{

User u = session.getUser();
if(!u.isAuthenticated() || !(permissionService.permissionsFor(u, file).contains(Permission.PublishDataset))) {
if(!u.isAuthenticated() || !u.isSuperuser()) {
//Shouldn't happen (choice not displayed for users who don't have the right permission), but check anyway
logger.warning("User: " + u.getIdentifier() + " tried to ingest a file");
JH.addMessage(FacesMessage.SEVERITY_WARN, BundleUtil.getStringFromBundle("file.ingest.cantIngestFileWarning"));
Expand Down Expand Up @@ -544,23 +544,29 @@ public String ingestFile() throws CommandException{
}

public String uningestFile() throws CommandException {

if (!file.isTabularData()) {
if(file.isIngestProblem()) {
User u = session.getUser();
if(!u.isAuthenticated() || !(permissionService.permissionsFor(u, file).contains(Permission.PublishDataset))) {
logger.warning("User: " + u.getIdentifier() + " tried to uningest a file");
//Shouldn't happen (choice not displayed for users who don't have the right permission), but check anyway
JH.addMessage(FacesMessage.SEVERITY_WARN, BundleUtil.getStringFromBundle("file.ingest.cantUningestFileWarning"));
return null;
}
file.setIngestDone();
file.setIngestReport(null);
//Ingest never succeeded, either there was a failure or this is not a tabular data file
User u = session.getUser();
if (!u.isAuthenticated() || !u.isSuperuser()) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I thought what you had here before was on purpose - to give the dataset author a way to remove an ingest failure report - no?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(those are only visible to the owners/authors, so not a big deal either way)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be nice if that's OK. (Probably less work for superusers as that warning bothers the authors.)
At QDR all of this is with the publish permission.

logger.warning("User: " + u.getIdentifier() + " tried to uningest a file");
// Shouldn't happen (choice not displayed for users who don't have the right
// permission), but check anyway
JH.addMessage(FacesMessage.SEVERITY_WARN,
BundleUtil.getStringFromBundle("file.ingest.cantUningestFileWarning"));
return null;
}
if (file.isIngestProblem()) {
file.setIngestDone();
file.setIngestReport(null);
} else {
JH.addMessage(FacesMessage.SEVERITY_WARN, BundleUtil.getStringFromBundle("file.ingest.cantUningestFileWarning"));
return null;
//Shouldn't happen - got called when there is no tabular data or an ingest problem
JH.addMessage(FacesMessage.SEVERITY_WARN,
BundleUtil.getStringFromBundle("file.ingest.cantUningestFileWarning"));
return null;
}
} else {
//Uningest command does it's own check for isSuperuser
commandEngine.submit(new UningestFileCommand(dvRequestService.getDataverseRequest(), file));
Long dataFileId = file.getId();
file = datafileService.find(dataFileId);
Expand All @@ -580,12 +586,11 @@ public String uningestFile() throws CommandException {
}
}
save();
//Refresh filemetadata with file title, etc.
// Refresh filemetadata with file title, etc.
init();
JH.addMessage(FacesMessage.SEVERITY_INFO, BundleUtil.getStringFromBundle("file.uningest.complete"));
return returnToDraftVersion();
}

}

private List<FileMetadata> filesToBeDeleted = new ArrayList<>();

Expand Down
9 changes: 9 additions & 0 deletions src/main/java/propertyFiles/Bundle.properties
Original file line number Diff line number Diff line change
Expand Up @@ -2203,6 +2203,15 @@ ingest.csv.lineMismatch=Mismatch between line counts in first and final passes!,
ingest.csv.recordMismatch=Reading mismatch, line {0} of the Data file: {1} delimited values expected, {2} found.
ingest.csv.nullStream=Stream can't be null.

file.ingest=Ingest
file.uningest=Uningest
file.ingest.alreadyIngestedWarning=This file has already been ingested
file.ingest.ingestInProgressWarning=Ingestion of this file is already in progress
file.ingest.cantIngestFileWarning=Ingest not supported for this file type
file.ingest.ingestQueued=Ingestion has been requested
file.ingest.cantUningestFileWarning=This file cannot be uningested
file.uningest.complete=Uningestion of this file has been completed

# editdatafile.xhtml

# editFilesFragment.xhtml
Expand Down