Add statistics pages #851

samuelcambien · 2020-08-28T13:14:56Z

References

https://github.com/DSpace/Rest7Contract/blob/main/statistics-reports.md
Fixes #731

Description

This PR implements the statistics pages for DSpace 7.

Instructions for Reviewers

List of changes in this PR:

I've added a StatisticsPageModule for all components related to the statistics pages. Added this module to load on the 'statistics' path.
I've added a generic StatisticsPage component to represent a statistics page.
I've added subclasses to this component for the site, community, collection and item statistics pages, with the reports to show on those pages. Added corresponding routes to the statistics page routing module.
I've added a context-sensitive link to the sidebar menu to the relevant statistics page.
I've added a StatisticsTableComponent to render a usage report as a table.

Checklist

This checklist provides a reminder of what we are going to look for when reviewing your PR. You need not complete this checklist prior to creating your PR (draft PRs are always welcome). If you are unsure about an item in the checklist, don't hesitate to ask. We're here to help!

My PR is small in size (e.g. less than 1,000 lines of code, not including comments & specs/tests), or I have provided reasons as to why that's not possible.
My PR passes TSLint validation using yarn run lint
My PR includes TypeDoc comments for all new (or modified) public methods and classes. It also includes TypeDoc for large or complex private methods.
My PR passes all specs/tests and includes new/updated specs or tests based on the Code Testing Guide.
If my PR includes new, third-party dependencies (in package.json), I've made sure their licenses align with the DSpace BSD License based on the Licensing of Contributions documentation.

lgtm-com · 2020-08-28T13:23:00Z

This pull request introduces 1 alert when merging 29553b5 into cc618eb - view on LGTM.com

new alerts:

1 for Unused variable, import, function or class

tdonohue

@samuelcambien : I gave this a quick code review today & tested it with the Backend PR (DSpace/DSpace#2782). Overall, the code looks OK, but I'm not able to get it working as expected (it's possible some of the issues are backend bugs...I'm looking into it).

The site-level statistics page is always empty for me (the page is entirely blank, with just a page heading). Even after browsing around to items, I cannot get anything to appear at http://localhost:4000/statistics/
The "Statistics" menu item (at the top of each page) appears to be caching its path & not updating as I browse from Site to Community to Collection to Item. If you start at the Site level and browse down, it always remains http://localhost:4000/statistics/ However, if I reload a Collection page in the browser, then that "Statistics" link is updated to the full Collection statistics path.
On Community, Collection and Item statistics the results shown are always zero. This might be a backend bug though, as it appears that is what is returned by my REST API even though I see successful viewevents sent each time I browse to a new object.

I've added a few more comments inline below.

tdonohue · 2020-09-04T16:26:06Z

src/app/+statistics-page/collection-statistics-page/collection-statistics-page.component.ts

+  templateUrl: '../statistics-page/statistics-page.component.html',
+  styleUrls: ['./collection-statistics-page.component.scss']
+})
+export class CollectionStatisticsPageComponent extends StatisticsPageComponent<Collection> {


Can we move all these new statistics components/pages under src/app/statistics-page? There's no need for the "+" in the path. See this long standing issue #327

Ok, renamed the directory

tdonohue · 2020-09-04T16:28:02Z

src/app/core/submission/usage-report-data.service.ts

+
+@Injectable()
+@dataService(USAGE_REPORT)
+export class UsageReportService extends DataService<UsageReport> {


Please add TypeDocs for this new class and its methods

Added typedocs

I think that this class has been wrongly created in core/submission folder, shouldn't it stay in the core/statistics folder ?

tdonohue · 2020-09-04T16:28:40Z

src/app/core/statistics/models/usage-report.model.ts

+
+@typedObject
+@inheritSerialization(HALResource)
+export class UsageReport extends HALResource {


Please add TypeDocs for this new class, and the Point interface in this same file

Added typedocs

samuelcambien · 2020-09-10T08:14:17Z

Hi Tim,

Thanks for your feedback.
As for the missing statistics (1. and 3.), I cannot reproduce this: with my backend I have statistics on all pages.
I guess these are not front-end issues.
As for 2., the statistics links are not properly updated indeed.

One way to solve this would be

to remove the 'compareArraysUsingIds()' function here:
https://github.com/DSpace/dspace-angular/blob/main/src/app/shared/menu/menu.component.ts#L82
and to re-write the menu effects to remove & add the persistent menu sections if they are resolved again, here:
https://github.com/DSpace/dspace-angular/blob/main/src/app/shared/menu/menu.effects.ts#L50

But I think this would decrease performance, so instead I simply renamed the menu id's so they are all unique :)

tdonohue

@samuelcambien : I've retested this today after I figured out a more reliable way to generate some viewevents statistics in the backend PR.

I'm still seeing the following issues:

The site-level statistics page is still always empty for me (the page is entirely blank, with just a page heading). I cannot get anything to appear at http://localhost:4000/statistics/ I even tried to generate a viewevent on the Site object itself by running this curl command: curl -X POST 'http://localhost:8080/server/api/statistics/viewevents' -H "Content-Type:application/json" -H "X-Forwarded-For:23.249.35.27" --data '{"targetId":"6d65c6a2-3fe7-44dd-bacb-79271257c35d", "targetType":"site"}' I still do not see any data on that main page, even though I can verify (via REST and other UI pages) that I have statistics generated.
The "Statistics" menu item (at the top of each page) appears to be caching its path & not updating as I browse from Site to Community to Collection to Item. I'm not sure of the best way to solve this issue, but I mentioned it to @artlowel , so hopefully he can give you some advice.

The Community/Collection/Item pages appear to work (after I generated some statistics locally). I now see report information pertaining to "Total Visits", "Total visits per month", "Top country views" and "Top city views".

As a sidenote, it looks like there's also a small merge conflict in this PR now.

codecov · 2020-09-23T14:32:11Z

Codecov Report

Merging #851 into main will increase coverage by 0.03%.
The diff coverage is 85.83%.

@@            Coverage Diff             @@
##             main     #851      +/-   ##
==========================================
+ Coverage   79.74%   79.77%   +0.03%     
==========================================
  Files         999     1008       +9     
  Lines       21875    21993     +118     
  Branches     2280     2291      +11     
==========================================
+ Hits        17444    17546     +102     
- Misses       3427     3443      +16     
  Partials     1004     1004

Impacted Files	Coverage Δ
src/app/core/core.module.ts	`60.00% <ø> (ø)`
src/app/navbar/navbar.component.ts	`56.25% <ø> (ø)`
...c/app/core/submission/usage-report-data.service.ts	`23.52% <23.52%> (ø)`
...age/statistics-table/statistics-table.component.ts	`80.00% <80.00%> (ø)`
src/app/shared/menu/menu.effects.ts	`97.77% <95.00%> (-2.23%)`	⬇️
...c/app/core/statistics/models/usage-report.model.ts	`100.00% <100.00%> (ø)`
...re/statistics/models/usage-report.resource-type.ts	`100.00% <100.00%> (ø)`
...stics-page/collection-statistics-page.component.ts	`100.00% <100.00%> (ø)`
...istics-page/community-statistics-page.component.ts	`100.00% <100.00%> (ø)`
...-statistics-page/item-statistics-page.component.ts	`100.00% <100.00%> (ø)`
... and 13 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a0414e1...c29ac36. Read the comment docs.

…d add id params

samuelcambien · 2020-09-24T07:23:27Z

Hi Tim,
Thanks for your feedback.

The site statistics page is probably empty for you because it is not supposed to display view events on the site object, instead it shows statistics for all items in the repository, similar to the same page in DSpace 6:

I've set up the backend branch locally, but I ran into an authorization issue on the site statistics page.
This is related to 2930, so I rebased the backend branch on the latest main to have this PR included.
Then I ran the db command mentioned here:

Afterwards, I was able to generate some item stats with the request you mentioned (but with uuid of an item).
After committing the solr statistics core (or waiting for 15 minutes) I got data on the site statistics page.

As for the caching links, I've checked with Art and he agreed that altering the ID's would be the best solution.
I've added an extra fix to append the uuid to the link id, so they would also be unique among items/collections/communities.
Rebased on the latest master to resolve conflicts.

atarix83

thanks @samuelcambien for this PR
Generally code looks good, I've only added an inline comment.
I've also tested new statistics pages and they seem to work properly.
I've only a comment, it should be nice if the statistic page show a loading spinner while retrieving statistics from server and if any the message No data available. What happens now is that the page show immediately the No data available message that is replaced by statistics later

atarix83 · 2020-09-24T12:37:34Z

src/app/core/submission/usage-report-data.service.ts

+
+@Injectable()
+@dataService(USAGE_REPORT)
+export class UsageReportService extends DataService<UsageReport> {


I think that this class has been wrongly created in core/submission folder, shouldn't it stay in the core/statistics folder ?

samuelcambien · 2020-09-29T12:48:05Z

Hi Guiseppe,
Thanks for your feedback.
I moved the data service and also added the ds-loading component.
The 'no data available' message is now only displayed when the reports have loaded.

tdonohue

👍 This looks good to me now. I also ran some tests and it appears to be working. (Note though that I hit bug DSpace/DSpace#2965 again in my testing...so, other testers need to be aware you must have a dbfile installed for stats to be logged). I expect we'll want to enhance these (very basic) stats reports, but they are a good initial version

atarix83

@samuelcambien thanks for addressed my feedback
however making more test, I probably found a bug. Indeed if I show statistics for a community and than trying to show statistics for other community I always get statistics of the first community I've visited. This is valid also between collections and items. Could you check it?

samuelcambien · 2020-10-07T12:31:04Z

Hi Guiseppe, thanks for pointing this out.
This was due to a problem in the MenuEffects resolveSubstitutions method, which caused the result to be stored in the route object.
Should be resolved by the latest commit.

atarix83

@samuelcambien thanks for the fix, it works properly now.
@tdonohue I have a suggestion that should be nice to improve in a follow-up PR.
Basically the breadcrumb in the statistics should include also the link of the related dspace objects, this should allow to easily navigate back to the resource for which statistics are being displayed

artlowel assigned samuelcambien Aug 28, 2020

artlowel added this to Needs Reviewers Assigned in DSpace 7 Beta 4 via automation Aug 28, 2020

artlowel added this to the 7.0beta4 milestone Aug 28, 2020

heathergreerklein requested review from atarix83 and tdonohue September 1, 2020 16:10

heathergreerklein moved this from Needs Reviewers Assigned to Under Review in DSpace 7 Beta 4 Sep 1, 2020

samuelcambien force-pushed the usage-reports branch from 29553b5 to 752ada3 Compare September 3, 2020 16:05

tdonohue requested changes Sep 4, 2020

View reviewed changes

samuelcambien force-pushed the usage-reports branch from 329bdc1 to cb3194c Compare September 10, 2020 07:26

tdonohue requested changes Sep 11, 2020

View reviewed changes

tdonohue mentioned this pull request Sep 11, 2020

DSpace (Solr) Statistics - Usage Reports DSpace/DSpace#2782

Merged

6 tasks

samuelcambien force-pushed the usage-reports branch 3 times, most recently from fbfd494 to 5a3a195 Compare September 23, 2020 14:01

samuelcambien force-pushed the usage-reports branch 3 times, most recently from c29ac36 to a7bf1ad Compare September 23, 2020 18:58

samuelcambien added 2 commits September 23, 2020 21:57

Add statistics pages

b7382de

Add statistics pages - feedback

e7e408d

samuelcambien force-pushed the usage-reports branch from a7bf1ad to c39b7ff Compare September 23, 2020 19:58

Add statistics pages - add support for parameters in menu sections an…

6ff9429

…d add id params

samuelcambien force-pushed the usage-reports branch from c39b7ff to 6ff9429 Compare September 23, 2020 21:28

samuelcambien requested a review from tdonohue September 24, 2020 07:37

atarix83 requested changes Sep 24, 2020

View reviewed changes

Add statistics pages - move data service and add ds-loading

b6ff936

samuelcambien requested a review from atarix83 September 29, 2020 12:49

tdonohue approved these changes Sep 30, 2020

View reviewed changes

tdonohue moved this from Under Review to Reviewer Approved in DSpace 7 Beta 4 Oct 1, 2020

tdonohue mentioned this pull request Oct 1, 2020

DS-4440 GDPR - Anonymize statistics feature DSpace/DSpace#2692

Merged

DSpace 7 Beta 4 automation moved this from Reviewer Approved to Under Review Oct 6, 2020

atarix83 requested changes Oct 6, 2020

View reviewed changes

Add statistics pages - fix link caching issue

d843230

samuelcambien requested a review from atarix83 October 7, 2020 12:31

DSpace 7 Beta 4 automation moved this from Under Review to Reviewer Approved Oct 7, 2020

atarix83 approved these changes Oct 7, 2020

View reviewed changes

tdonohue merged commit 5bf6d0f into DSpace:main Oct 7, 2020

DSpace 7 Beta 4 automation moved this from Reviewer Approved to Done Oct 7, 2020

tdonohue mentioned this pull request Apr 19, 2021

Item-level statistics pages no longer working #1116

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add statistics pages #851

Add statistics pages #851

samuelcambien commented Aug 28, 2020 •

edited by artlowel

lgtm-com bot commented Aug 28, 2020

tdonohue left a comment •

edited

tdonohue Sep 4, 2020

samuelcambien Sep 10, 2020

tdonohue Sep 4, 2020

samuelcambien Sep 10, 2020

atarix83 Sep 24, 2020

tdonohue Sep 4, 2020

samuelcambien Sep 10, 2020

samuelcambien commented Sep 10, 2020

tdonohue left a comment •

edited

codecov bot commented Sep 23, 2020 •

edited

samuelcambien commented Sep 24, 2020

atarix83 left a comment

atarix83 Sep 24, 2020

samuelcambien commented Sep 29, 2020

tdonohue left a comment

atarix83 left a comment

samuelcambien commented Oct 7, 2020

atarix83 left a comment

Add statistics pages #851

Add statistics pages #851

Conversation

samuelcambien commented Aug 28, 2020 • edited by artlowel

References

Description

Instructions for Reviewers

Checklist

lgtm-com bot commented Aug 28, 2020

tdonohue left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

samuelcambien commented Sep 10, 2020

tdonohue left a comment • edited

Choose a reason for hiding this comment

codecov bot commented Sep 23, 2020 • edited

Codecov Report

samuelcambien commented Sep 24, 2020

atarix83 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

samuelcambien commented Sep 29, 2020

tdonohue left a comment

Choose a reason for hiding this comment

atarix83 left a comment

Choose a reason for hiding this comment

samuelcambien commented Oct 7, 2020

atarix83 left a comment

Choose a reason for hiding this comment

samuelcambien commented Aug 28, 2020 •

edited by artlowel

tdonohue left a comment •

edited

tdonohue left a comment •

edited

codecov bot commented Sep 23, 2020 •

edited