Added new data query option "allow_past"

[stelfrag]

Summary

Adds a new query option allow_past for the data endpoint that will allow the after parameter to extend before the first timestamp stored. The entries that do not exist will be returned as null

Component Name

database

Test Plan

• Start agent will no history (e.g. if using dbengine memory mode, delete the dbengine files)
• Run a query such as:
  /api/v1/data?chart=system.cpu
  • This should return just a few entries (the number of seconds the agent is up and running)
• Run the query
  /api/v1/data?chart=system.cpu&options=allow_past
  • You should get 600 entries returned, most of them null

[thiagoftsm]

Hi @stelfrag ,

Please, can you also change the swagger file and yaml file with this PR for we do not have options without documentation?

Thank you!

[stelfrag]

Hi @stelfrag ,

Please, can you also change the swagger file and yaml file with this PR for we do not have options without documentation?

Thank you!

good point !!! 👍

[thiagoftsm]

Run a query such as:
/api/v1/data?chart=system.cpu

    This should return just a few entries (the number of seconds the agent is up and running)

Run the query
/api/v1/data?chart=system.cpu&options=allow_past

    You should get 600 entries returned, most of them null

@stelfrag I tested now with memory mode = ram and memory mode = dbengine, for this last I removed everything inside /var/cache/netdata/dbengine before to test, and I could not see the results that you described.

I executed the following requests to simplify the comparison:

$ curl -k -o without.json https://localhost:19999/api/v1/data?chart=system.cpu ; curl -k -o with.json https://localhost:19999/api/v1/data?chart=system.cpu&options=allow_past
$ wc -l with*

[stelfrag]

Run a query such as:
/api/v1/data?chart=system.cpu

    This should return just a few entries (the number of seconds the agent is up and running)

Run the query
/api/v1/data?chart=system.cpu&options=allow_past

    You should get 600 entries returned, most of them null

@stelfrag I tested now with memory mode = ram and memory mode = dbengine, for this last I removed everything inside /var/cache/netdata/dbengine before to test, and I could not see the results that you described.

I executed the following requests to simplify the comparison:

$ curl -k -o without.json https://localhost:19999/api/v1/data?chart=system.cpu ; curl -k -o with.json https://localhost:19999/api/v1/data?chart=system.cpu&options=allow_past
$ wc -l with*

Ah, yes I have been tricked quite a few times by this.

Please escape the & because you sent the command to the background 😄

[thiagoftsm]

After to add quotes for curl 👀 , everything worked as expected.
Thank you @stelfrag !