[DS-4301] Ensure that DSpace REST Report capabilities are part of the DSpace 8 REST API

[dspace-bot]

Imported from JIRA [DS-4301] created by terrywbrady

The tools are described here.

 

https://wiki.duraspace.org/display/DSDOC6x/REST+Based+Quality+Control+Reports

 

These tools allow for very specific validation of metadata fields: field is present, field is missing, field matches a regular expression.

Legacy REST endpoints used by the report tools

• /rest/filtered-items - construct a query that applies several metadata query statements (in HQL).  The results can be further processed with filters.
  • The query portion of this endpoint is the most important feature of these tools.  If SOLR queries could provide an accurate validation of metadata rules for specific fields, that would probably be the best solution.  Otherwise, a set of query expressions will need to be converted to HQL.  That is the current implementation.
• /rest/hierarchy - retrieve a full dump of the community/collection hierarchy - used to construct drop down lists
  • The REST contract would have endpoints to return this data, but it they may not provide a client-friendly representation of the whole hierarchy.
• /rest/filters - a list of interesting attributes of items to be filtered.  Currently these filters are applied in code rather than in HQL/SQL.
  • Many filters look at bitstream properties and bitstream permissions.  It was difficult to convert all of these filters into HQL expressions.  Depending on the recommended approach for /rest/filtered-items, we should consider if these could be expressed as a SOLR query or HQL query.  Otherwise, the filter object may be useful to retain.
• /rest/filtered-collections - count the items in a collection that match a set of filters.
  • Depending on the filter implementation chosen above, this endpoint should be recreated.

Legacy endpoint documentation

• https://wiki.duraspace.org/display/DSDOC6x/REST+Reports+-+Summary+of+API+Calls#RESTReports-SummaryofAPICalls-GET/rest/filtered-items-Retrieveasetofitemsbasedonametadataqueryandasetoffilters
• https://terrywbrady.github.io/restReportTutorial/intro

Rest Report Client

Once the endpoint code is in place, it should be fairly simple to recreate the 2 rest report client pages.  The updated client should take advantage of the authentication features of the new REST 7API.

The REST Report Client displays the results of these queries in tabular form.  That table can be directly output and used by the bulk metadata update process.  Additionally, some high level bitstream metadata can appear in the report results.

Entity Support

 The queries will need to evolve to support queries on DSpace entity metadata.

[dspace-bot]

Is related to: DS-4303

[dspace-bot]

alaw said:

I have used these reporting tools and found them very helpful for cleaning up metadata and checking the status of bitstreams. For instance, using the checksum column, I found 200 ETD file pairs in our collection of ~30,000 that had different filenames but were actually duplicates, in addition to many duplicate pairs with identical filenames. Without Reporting Tools, bitstream data is much harder to obtain. We used the Reports to discover items that we had not realized were missing files. I would very much like the Reporting Tools preserved in later versions of DSpace.

[dspace-bot]

pbecker said:

I know at least two German repositories that uses the RESTvreport tools on a regular base. Both are larger installations and at least one of the already contributed back to DSpace. Would be great to see this being continued and supported in the future.

[dspace-bot]

tdonohue said:

Just a note here, one of the primary reasons for keeping the old REST API (v4-6) around in DSpace 7 is to continue support of these reports in DSpace 7 (and the future).  My hope is that someone(s) would adapt these reporting tools to use the (new) DSpace 7 REST API prior to the DSpace 8 release (or even just after 7.0 is out).  Technically, the tool need not be packaged as part of the official REST API itself, as this tool is a client of the REST API. Therefore, it could be made available as a separate client project, or even as part of the Angular UI (if it is more easily adapted there).

So, this is all to say that I agree with others that these reporting tools seem useful to continue to maintain. However, some effort will have to be made to adapt them to the new REST API.

[dspace-bot]

mwood said:

I think that that is the purpose of this Issue:  to port specific capabilities of "old REST" into "new REST" so that the above-mentioned reporting package can be ported to "new REST", removing a significant roadblock to retiring the "old REST" code.

DSpace 7 will have the same situation w.r.t. REST that DSpace 6 has w.r.t. GUI:  we have two to maintain, and we'd be better off with only one.  We should consider making REST feature parity and deprecation of "old REST" goals for DSpace 8.  That, I think, is what this Issue is really saying, though it is expressed in terms of a specific REST client.

[dspace-bot]

tdonohue said:

If the goal of this ticket is feature "parity" between the old REST API and new REST API, then I'd recommend we create separate (perhaps sub-) tickets listing the features of the old REST API that are not yet available in the new REST API.  At this time, it's unclear to me whether any such gaps exist (and whether those gaps are easily solvable prior to DSpace 7 or require extra effort/work and should be scheduled for DSpace 8)

To Mark's point, the old REST API will be deprecated in DSpace 7.  So, two REST APIs will only exist in that DSpace 7.x release, allowing developers/institutions using the old REST API to migrate all tooling to the new REST API.  As of DSpace 8, there will only be a single REST API (the new one).

[pilasou]

Hi @tdonohue, we will try to advance this issue. @jeffmorin will be working on it. We will focus on Legacy REST endpoints used by the report tools but not on the new Entity Support for now. Since Entity Support involves working on a new functionality (i.e. not porting an existing one), I suggest creating a new ticket for it specifically. Moreover, a functional specification will be needed for any willing contributor to properly work on it.

[tdonohue]

@pilasou : There is no requirement in this issue ticket to update the existing reporting tools to also report on Entities. I agree that could be future work, as that's not been defined at all yet. The only requirement in this ticket is to ensure the old reports don't "die" when the old REST API is removed from DSpace (in v8). So, I agree that the reporting tools do not need to be updated beyond fixing them to work with the new REST API... any additional improvements can always come later as needed.

If you or @jeffmorin have additional questions, let us know. As usual, we are open to suggestions of the "best way" to resolve this ticket, the end goal is just to ensure the old reports are retained in some manner (how they are retained is open for suggestion/discussion....it could be porting them over mostly "as-is" to the new REST API or it could even be moving them into the Angular UI, or some other solution I haven't thought of)

[jeffmorin]

I created a pull request for the REST specification of the first report (Filtered Collections): DSpace/RestContract#202.