Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add inspection modes to Python client library #1418

Merged
merged 4 commits into from
Apr 25, 2024

Conversation

rzats
Copy link
Contributor

@rzats rzats commented Apr 18, 2024

Closes #1321.

Summary:

Adds two debug modes to the Python client library (JS and R clients TBD):

  • debug logs info about HTTP requests and responses.
  • sandbox prevents any HTTP requests from actually executing, allowing users to test a long-running or expensive script without any actual server load.

Prerequisites:

  • Unless it is a documentation hotfix it should be merged against the dev branch
  • Branch is up-to-date with the branch to be merged with, i.e. dev
  • Build is successful
  • Code is cleaned up and formatted

Copy link
Collaborator

@melange396 melange396 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pretty nice!

Could you, instead of or in addition to being in the main log messages, put all the variables reported into their own keyword arguments? That will allow for easier post-processing, among other things.

@@ -71,9 +78,19 @@ def _list(values):
def _request_with_retry(endpoint, params={}):
"""Make request with a retry if an exception is thrown."""
request_url = f"{Epidata.BASE_URL}/{endpoint}/"
if Epidata.debug:
Epidata.logger.info(f"Sending GET request to URL: {request_url} | params: {params} | headers: {_HEADERS}")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should probably log Epidata.auth here too since its used in the request

Copy link
Contributor

@dshemetov dshemetov Apr 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggest(non-blocking): since we're using structlog, you can feed params in like a dict and it will handle the formatting, like

Epidata.logger.info("Sending GET request", url=request_url, params=params, headers=_HEADERS)

I would play with this and make sure the output looks reasonable (since I haven't done it myself), but since params and _HEADERS are probably dicts, their repr() method should make them look nice too. And then structlog will render it all in a consistent JSON.

src/client/delphi_epidata.py Outdated Show resolved Hide resolved
src/client/delphi_epidata.py Outdated Show resolved Hide resolved
src/client/delphi_epidata.py Outdated Show resolved Hide resolved
dshemetov
dshemetov previously approved these changes Apr 19, 2024
Copy link
Contributor

@dshemetov dshemetov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small suggestion about log message formatting, otherwise looks good 👍

@@ -71,9 +78,19 @@ def _list(values):
def _request_with_retry(endpoint, params={}):
"""Make request with a retry if an exception is thrown."""
request_url = f"{Epidata.BASE_URL}/{endpoint}/"
if Epidata.debug:
Epidata.logger.info(f"Sending GET request to URL: {request_url} | params: {params} | headers: {_HEADERS}")
Copy link
Contributor

@dshemetov dshemetov Apr 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggest(non-blocking): since we're using structlog, you can feed params in like a dict and it will handle the formatting, like

Epidata.logger.info("Sending GET request", url=request_url, params=params, headers=_HEADERS)

I would play with this and make sure the output looks reasonable (since I haven't done it myself), but since params and _HEADERS are probably dicts, their repr() method should make them look nice too. And then structlog will render it all in a consistent JSON.

dshemetov
dshemetov previously approved these changes Apr 24, 2024
Copy link
Contributor

@dshemetov dshemetov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

melange396
melange396 previously approved these changes Apr 24, 2024
Copy link
Collaborator

@melange396 melange396 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add in the auth logging, and then i think this is done!

src/client/delphi_epidata.py Outdated Show resolved Hide resolved
Co-authored-by: george <george.haff@gmail.com>
@rzats rzats dismissed stale reviews from melange396 and dshemetov via aaff161 April 25, 2024 13:33
Copy link

sonarcloud bot commented Apr 25, 2024

Quality Gate Passed Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

@rzats
Copy link
Contributor Author

rzats commented Apr 25, 2024

@melange396 @dshemetov re-requesting review as the latest commit invalidated the "approved" checkmarks :)

@melange396 melange396 merged commit 988f10a into dev Apr 25, 2024
7 checks passed
@melange396 melange396 deleted the rzatserkovnyi/client-inspection branch April 25, 2024 13:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add "inspection" mode to client libraries
3 participants