Build out API for creating report sections, tiles, and trend graphs

[pippinsplugins]

As part of the 3.0 development push, we're going to create a new API that allows core and extensions to easily create new report options.

Tracking PR	Project
#6273	3DD™: Reporting API Reboot

This API will comprise three primary pieces:

Infrastructure

• Create a Registry API #6253 – API to register (any) stuff
• Create a Registry API #6253 – API to handle exceptions
• Introduce a class for initializing the new Reports API #6260 – Concrete class to initialize the Reports API

Report (tabs)

• Create an API for registering reports (tabs) #6254 – API to register report tabs (PR: Issue/6254 – Reports Tab API #6263)
• Create an API to build and render reports #6284 – Report object (PR: issue/6284 – Report object API #6289)
• Create an API to build and render reports #6284 – API to build and render reports (PRs: issue/6284 – Report object API #6289, issue/6284 – Render reports #6329)
• Integrate Carbon to simplify date and lookback date filtering in Reports #6334 – Integrate the Carbon library to simplify date filtering (PR: Issue/6334 – Utils\Date + Carbon #6353)
• Create an API for registering filters against reports #6257 – API to register filters against report tabs (PR: Issue/6257 – Endpoint Filters API #6361)
• Reports API: Output reports using a callback with legacy action support #6360 – Deprecate old report view/tab display actions using a callback (PR: Issue/6360 – Report display method and callback #6371)
• changelog for 2.8.14 #6212 – Audit test coverage for Reports API Reboot pre-merge (PR: Issue/6512 – Add missing test coverage for the Reports API reboot #6520)

Endpoints

• Introduce an Endpoint API #6280 – API for Endpoints (PR: issue/6280 – Endpoint API #6283)
• Add tile support to the Endpoints API #6285 – Add tile support to the Endpoints API (PR: Issue/6285 – Tile endpoint support #6313)
• Add chart support to the Endpoints API #6287 – Add chart support to the Endpoints API
• Add table support to the Endpoints API #6286 – Add table support to the Endpoints API (PR: Issue/6286 – Table endpoint support #6339)
• Add pie chart support to the chart endpoint #6511 – Add pie chart support to the chart endpoint (PR: Issue/6511 – Pie/doughnut chart support #6515)

Nice to Have

• Expose reports endpoints via REST #6281 – Expose endpoints via REST

The reports screen from AffiliateWP represents a fairly good view of what this API will ultimately provide:

[DrewAPicture]

PR submitted to @mindctrl for review on #6253.

[DrewAPicture]

The empty EDD_Reports class has been merged to issue/6236 via #6260. And so it goes.

[DrewAPicture]

It's worth noting that while we're in the process of moving the Logs tab from Reports to Tools, this tab largely comprises graphs that will (likely) be directly integrated into new Reports views, so may actually be able to just completely deprecate the Logs tab altogether at the end of this thing.

[DrewAPicture]

EDD_Registry was merged via #6253.

[DrewAPicture]

The Logs tab was moved to Tools in #6265.

[DrewAPicture]

Added PR for the master tracking issue (#6273). This is obviously going to be merged last, but all sub-issues are using issue/6236 as a "master" branch, so this PR can be used to get a 10,000 ft view of progress on the Reports API reboot.

[DrewAPicture]

Decided to consolidate the effort between #6254 and #6255 into the PR for just #6254.

[DrewAPicture]

Added to-do items for the following:

• Introduce an Endpoint API #6280 – Introduce an Endpoint API
• Expose reports endpoints via REST #6281 – Expose reports endpoints via REST
• Create an API to build and render reports #6284 – Create an API to build and render reports

The REST issue is still a maybe and lower priority than any of the others. But it would be nice to have.

[DrewAPicture]

Merged the Endpoint API in #6280.

[DrewAPicture]

Merged #6329 containing the bulk of the reports rendering logic for #6284.

[DrewAPicture]

Merged #6339 to add table endpoint support.

[DrewAPicture]

Integrated the Carbon library and introduced a variety of date functions and helpers in #6334.

[DrewAPicture]

Merged #6361. to introduce an API for registering and using filters.

[DrewAPicture]

Merged the PR for #6287 to introduce chart endpoint support.

[DrewAPicture]

#6515 was merged for #6511 to add pie chart endpoint support.

[DrewAPicture]

#6520 was merged for #6512 bringing this up to pretty much full test coverage!

[DrewAPicture]

The Reports API Reboot is now feature complete and ready for review! 🎉

[sunnyratilal]

@easydigitaldownloads/core-team Could we please have some testing done on this PR as it is pivotal for the next phase of 3DD where this new API will be extensively used.

@DrewAPicture has done phenomenal work here.

The Reports page now separates reports by vertical tabs. Each report can consist of any combination of three endpoint types: tiles, tables, and charts.

Please test with this example plugin that Drew created demonstrating how each of the three aforementioned endpoints will work (this will also provide a good example of how all the new APIs are used to create reports): https://gist.github.com/DrewAPicture/69acc0a74af184f11443687802ac89a4

There are two ways reports can be created using the new API: one is the using the way above and the other is by directly the functions: https://gist.github.com/DrewAPicture/d0717133deb5c1c7b569dfdbdd730676. The first requires developers to wrap their code in a try/catch, the second does not. However, both will still throw exceptions if invalid data is passed. Exceptions will automatically be logged to the debug log if Debug Mode is enabled.

To Test

1. Check Reports page loads all of the existing EDD reports and displays existing charts
2. Install the Example Report plugin and ensure that the endpoints are displayed correctly.

Note:
Deprecated PHP notices are being thrown and this is intended. These notices will begin to disappear once we recreate the reports using the new APIs.

[DrewAPicture]

It's worth noting also that this PR is largely a representation of the reports API internals only, that is, I fully expect the implementation team to build outer layers for the registering and configuring of reports, endpoints, and display mechanisms in EDD.

So please try to keep in mind that this really just represents the first part of the reports overhaul – the guts; none of the existing core reports have been converted to use any of the new APIs (the implementation team will do that).

A few functions and hooks have been deprecated, and as @sunnyratilal alluded, will throw expected notices. The styles and UIs and interactions of the elements will likely change once the implementation team gets in there.

Example of expected notices (in lieu of newly implemented core reports):

PHP Notice:  The edd_report_views hook is <strong>deprecated</strong> since Easy Digital Downloads version 3.0! Use the edd_reports_get_tabs hook instead. in /path/to/wp/wp-content/plugins/easy-digital-downloads/includes/misc-functions.php on line 735
PHP Notice:  The edd_report_views hook is <strong>deprecated</strong> since Easy Digital Downloads version 3.0! Use the edd_reports_get_tabs hook instead. in /path/to/wp/wp-content/plugins/easy-digital-downloads/includes/misc-functions.php on line 735
PHP Notice:  The edd_report_views hook is <strong>deprecated</strong> since Easy Digital Downloads version 3.0! Use the edd_reports_get_tabs hook instead. in /path/to/wp/wp-content/plugins/easy-digital-downloads/includes/misc-functions.php on line 735
PHP Notice:  The edd_report_views hook is <strong>deprecated</strong> since Easy Digital Downloads version 3.0! Use the edd_reports_get_tabs hook instead. in /path/to/wp/wp-content/plugins/easy-digital-downloads/includes/misc-functions.php on line 735
PHP Notice:  The edd_reports_view_earnings hook is <strong>deprecated</strong> since Easy Digital Downloads version 3.0! Use the \EDD\Reports\add_report hook instead. in /path/to/wp/wp-content/plugins/easy-digital-downloads/includes/misc-functions.php on line 735
PHP Notice:  edd_get_report_dates is <strong>deprecated</strong> since Easy Digital Downloads version 3.0! Use \EDD\Reports\get_dates_filter instead. in /path/to/wp/wp-content/plugins/easy-digital-downloads/includes/misc-functions.php on line 535
PHP Notice:   in /path/to/wp/wp-content/plugins/easy-digital-downloads/includes/misc-functions.php on line 536
PHP Notice:  edd_get_reporting_view is <strong>deprecated</strong> since Easy Digital Downloads version 3.0 with no alternative available. in /path/to/wp/wp-content/plugins/easy-digital-downloads/includes/misc-functions.php on line 539
PHP Notice:   in /path/to/wp/wp-content/plugins/easy-digital-downloads/includes/misc-functions.php on line 540

The fact that the core reports still display is merely an indicator that backward compatibility has been taken into account for display purposes only.

[pippinsplugins]

@DrewAPicture do I need to do anything specific to support date filters? I see the date filters in the existing and new reports (using the example report provided) but they do not function. Is the example missing a key piece or is that a bug?

[spencerfinnell]

Noticed some tabs are outputting multiple date selectors. This causes the following JS to fail:

https://github.com/easydigitaldownloads/easy-digital-downloads/blob/c3a813cc5021409b295260a8edb761a7bbe9b5b8/assets/js/admin-scripts.js#L992-L1002

The multiple IDs on the page are invalid (technically) and causes only the first instance of the HTML to use the JS. The ID should be switched to a class selector if multiple will appear on the page.

Class selectors are slower but it would require looping through all instances of the ID otherwise, which is probably just as slow.

Also I think the previously used .show() and .hide() functions make more sense here as when the items are hidden they are not relevant to screen readers either.

Something like:

// toggle extended date options
$( '.edd-graphs-date-options' ).change( function() {
  var $this = $( this );
  var date_range_options = $this.closest( '.edd-date-range-options' );

  return date_range_options.toggle( 'other' === $this.val() );
} );

[DrewAPicture]

Just demonstrated the date filter saving state and adjusting a chart endpoint in person for @pippinsplugins so maybe something weird with his setup isn't working.

@spencerfinnell Noted. Probably can just switch to a class and each it.

[DrewAPicture]

Also I think the previously used .show() and .hide() functions make more sense here as when the items are hidden they are not relevant to screen readers either.

@spencerfinnell If they're UI elements, they're relevant. display:none completely hides the elements and unless you notify screen readers when you're showing and hiding elements with JS through something like wp.a11y.speak() every time, then those visual interactions are pretty much a complete mystery to screen reader users. Describing those fields via an aria-describedby or legend would probably be useful, regardless.

[DrewAPicture]

Merged #6273 to bring in the entirety of the new Reports API internals 🎉

New issues for anything that comes up in implementation.

[DrewAPicture]

Example report:

[spencerfinnell]

@DrewAPicture

If they're UI elements, they're relevant. display:none completely hides the elements and unless you notify screen readers when you're showing and hiding elements with JS through something like wp.a11y.speak() every time, then those visual interactions are pretty much a complete mystery to screen reader users. Describing those fields via an aria-describedby or legend would probably be useful, regardless.

But the fields are being hidden because they are not relevant to the other filters (and cannot be modified by mouse, keyboard, or voice). Having them visible to screen readers when using a range other than "Custom" would cause readers to announce them when they are not needed and are not able to be modified.

I'm not sure the best way to announce when they have appeared -- probably wp.a11y.speak() like you said.

[DrewAPicture]

Overview

The Reports API Reboot represents the next higher level of modularization for reports. The entire idea of this reboot is to break the data represented in a "report" into logical chunks, make it possible to build new or rebuild old reports using those chunks, and refresh how those chunks can be visualized

Think of the idea of a new "report" as a way to containerize data points (endpoints) in a structured way. A report contains endpoints, which each contain view definitions, and sometimes those view definitions contain datasets and manifests. Basically speaking, the reports API reboot takes what is currently hard-coded, modularizes it, and sets up a mechanism through which data is retrieved just in time for display.

Getting Registered

EDD 3.0 introduces the concept of a "registry". Items get added to a registry (in memory) and then can be retrieved globally for various purposes.

And there can be any number of independent registries created. The new Reports API introduces two registries, one each for reports and endpoints that will be consumed by those registered reports, respectively.

Moving forward, it's important to draw a distinction between what we think of as a "reports tab" and a report, because a "reports tab" represents the visualization of a "report" which is now a structured data object. More on that below.

Reports and endpoint registration are handled separately for reasons that will be explained later, but the main thing to consider is that reports and endpoints are added to registries to kill two birds with one stone:

1. Get all of the needed data together in one place for global access
2. Handle error checking and validation on the way in

The reason error checking and validation happens during registration, is that the registry data needs to be in a very specific format when it is later pulled out and built into structured data objects. If there's a problem on the way in, there is a whole bunch of internal logic in place to not only check for errors, but report those exceptions to the debug log (yay making support easier!).

Great pains have been taken to ensure that errors at any level of the API can be logged and tracked down if debug mode is turned on.

Reports and Endpoints and Views, oh my!

Let's talk a little bit more about terminology for a minute. For the purposes of the new reports API, a "report" is a structured data object representing all of the data that might be displayed with it. An "endpoint" is a structured data object leveraged by a report to display a piece of data. A view defines one or more ways an endpoint can be visualized.

To see this in action, check this sample plugin.

Here, two reports are registered in the register_report() method, each defining endpoints they would like to surface via several different "view groups", i.e. tiles, charts, or tables.

In the register_endpoints() method, endpoints are registered. Notice that each endpoint defines views through which that particular data point can be visualized. Each view definition contains its own display and data callbacks.

In comparing the report and endpoint registrations, notice the correlation between the view groups in the 'endpoints' attribute of the report and the view definitions in the 'views' attribute of the endpoint. tile:tiles as chart:charts as table:tables.

One Endpoint, Multiple Views

Basically, any one endpoint can register view definitions for any valid view type, making the data reusable in multiple contexts and even multiple different reports, thus why reports and endpoints are registered separately.

For example, earnings data could be expressed as as a tile (single piece of data), a table (data table), and/or a chart (pie/doughnut chart or line/bar graph). An endpoint simply needs to register at least one view definition to be valid.

Reports leverage view groups to determine whether registered endpoints contain compatible view definitions. If an endpoint registers 'tile' and 'table' views and a report calls for that endpoint using the 'charts' group, it's going to fail silently.

Structured Data

When data in the reports and endpoints registries is later retrieved, API functionality builds that data into structured data objects.

Just as you could conceivably hit a REST endpoint and get back JSON data in the form of a response, Report and Endpoint objects represent data, but in a much richer format than just raw data: the response is a fully-qualified data object decorated with helper methods and error checking.

The API doesn't currently contain actual REST endpoints, but it's built in a way that making that possible in the future would be fairly straightforward. See #6281.

Here is a basic rundown of what happens when a reports tab is loaded

1. A report tab starts to display
2. It calls for a specific, registered report
3. The report is retrieved from the registry and built into a Report object
4. At this time, all endpoints whitelisted for the report are looped through and Endpoint objects are built for each of them, as long as they contain valid view definitions and don't trigger any errors. If one or both conditions are not met, errors are logged to the Report object and that specific Endpoint object is not built.
5. Any whitelisted, valid filters are displayed
6. The report display callback loops through those rich endpoint objects and triggers their display methods
7. At this time, the data callbacks are finally fired, just in time for display – more on this below
8. The report is displayed with all of its endpoints.

If you note in step 7 above, the data callbacks don't get called until the Endpoint object display callbacks are called. This is what makes the new reports API so much faster: All of the data in the registry is static, and all of the data in the Endpoints is static right up until it's time to display it and then it's retrieved just in time.