Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: upgrade ZIO http client to improve performance #850

Merged
26 commits merged into from
Jan 25, 2024

Conversation

ghost
Copy link

@ghost ghost commented Jan 18, 2024

Overview

Previous to this PR, when using ZIO HTTP RC2 - we saw a very long delay when making HTTP requests (>2 seconds for each request) - we identified that this was not only an issue with the version of ZIO HTTP we were using but also how it was managing and shuting down the ZIO HTTP stack - essentially every request was starting and stopping the full stack.

After, with upgrading to ZIO HTTP RC4 and refactoring the ZIO layer composition for scope and client, we now see requests completing within 100 ms as the full stack doens't shutdown with every request

Also included in this PR are several improvements to the performance testing framework for error handling when polling background jobs and for managing exceptions when deserialising JSON - previously, any exception thrown was not counted as a failure in the test execution and dropped - now - metrics are available for when the background job does not complete a state transition within the given number of polls

Fixes ATL-6318

Checklist

My PR contains...

  • No code changes (changes to documentation, CI, metadata, etc.)
  • Bug fixes (non-breaking change which fixes an issue)
  • Improvements (misc. changes to existing features)
  • Features (non-breaking change which adds functionality)

My changes...

  • are breaking changes
  • are not breaking changes
  • If yes to above: I have updated the documentation accordingly

Documentation

  • My changes do not require a change to the project documentation
  • My changes require a change to the project documentation
  • If yes to above: I have updated the documentation accordingly

Tests

  • My changes can not or do not need to be tested
  • My changes can and should be tested by unit and/or integration tests
  • If yes to above: I have added tests to cover my changes
  • If yes to above: I have taken care to cover edge cases in my tests

ports:
- "9995:9187"
environment:
- DATA_SOURCE_NAME=postgresql://postgres:postgres@holder-db:5432/postgres?sslmode=disable

Check failure

Code scanning / SonarCloud

PostgreSQL database passwords should not be disclosed

<!--SONAR_ISSUE_KEY:AY0dVG_LK8zoueQutewS-->Make sure this PostgreSQL database password gets changed and removed from the code. <p>See more on <a href="https://sonarcloud.io/project/issues?id=input-output-hk_atala-prism-building-blocks&issues=AY0dVG_LK8zoueQutewS&open=AY0dVG_LK8zoueQutewS&pullRequest=850">SonarCloud</a></p>
ports:
- "9996:9187"
environment:
- DATA_SOURCE_NAME=postgresql://postgres:postgres@issuer-db:5432/postgres?sslmode=disable

Check failure

Code scanning / SonarCloud

PostgreSQL database passwords should not be disclosed

<!--SONAR_ISSUE_KEY:AY0dVG_LK8zoueQutewT-->Make sure this PostgreSQL database password gets changed and removed from the code. <p>See more on <a href="https://sonarcloud.io/project/issues?id=input-output-hk_atala-prism-building-blocks&issues=AY0dVG_LK8zoueQutewT&open=AY0dVG_LK8zoueQutewT&pullRequest=850">SonarCloud</a></p>
Copy link
Contributor

github-actions bot commented Jan 18, 2024

Unit Test Results

 85 files  ±0   85 suites  ±0   19m 21s ⏱️ -41s
719 tests ±0  711 ✅ ±0  8 💤 ±0  0 ❌ ±0 
725 runs  ±0  717 ✅ ±0  8 💤 ±0  0 ❌ ±0 

Results for commit 53ae046. ± Comparison against base commit 529a9e8.

♻️ This comment has been updated with latest results.

Copy link
Contributor

github-actions bot commented Jan 18, 2024

Integration Test Results

12 files  +1  12 suites  +1   2s ⏱️ ±0s
25 tests +1  25 ✅ +1  0 💤 ±0  0 ❌ ±0 
26 runs  +1  26 ✅ +1  0 💤 ±0  0 ❌ ±0 

Results for commit 53ae046. ± Comparison against base commit 529a9e8.

♻️ This comment has been updated with latest results.

@ghost ghost changed the title Feature/http client performance fix feat(prism-agent): upgrade ZIO http client to improve performance Jan 18, 2024
@atala-dev atala-dev added the ci label Jan 24, 2024
@ghost ghost changed the title feat(prism-agent): upgrade ZIO http client to improve performance feat: upgrade ZIO http client to improve performance Jan 24, 2024
davepoltorak and others added 18 commits January 24, 2024 15:28
Signed-off-by: David Poltorak <david.poltorak@iohk.io>
Signed-off-by: David Poltorak <david.poltorak@iohk.io>
Signed-off-by: David Poltorak <david.poltorak@iohk.io>
Signed-off-by: David Poltorak <david.poltorak@iohk.io>
…ance

Signed-off-by: David Poltorak <david.poltorak@iohk.io>
Signed-off-by: David Poltorak <david.poltorak@iohk.io>
Signed-off-by: David Poltorak <david.poltorak@iohk.io>
…requests

Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
…e) to agent's configuration file

Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
…f the active ZIO Scope

Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
Signed-off-by: David Poltorak <david.poltorak@iohk.io>
Signed-off-by: David Poltorak <david.poltorak@iohk.io>
Signed-off-by: David Poltorak <david.poltorak@iohk.io>
Signed-off-by: David Poltorak <david.poltorak@iohk.io>
Signed-off-by: David Poltorak <david.poltorak@iohk.io>
Signed-off-by: David Poltorak <david.poltorak@iohk.io>
Signed-off-by: David Poltorak <david.poltorak@iohk.io>
Signed-off-by: David Poltorak <david.poltorak@iohk.io>
Signed-off-by: David Poltorak <david.poltorak@iohk.io>
davepoltorak and others added 3 commits January 25, 2024 09:45
Signed-off-by: David Poltorak <david.poltorak@iohk.io>
Signed-off-by: David Poltorak <david.poltorak@iohk.io>
…error case

Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
@bvoiturier bvoiturier force-pushed the feature/http-client-performance-fix branch from 980ad1d to 53ae046 Compare January 25, 2024 11:19
Copy link

sonarcloud bot commented Jan 25, 2024

Quality Gate Passed Quality Gate passed

The SonarCloud Quality Gate passed, but some issues were introduced.

2 New issues
0 Security Hotspots
No data about Coverage
0.1% Duplication on New Code

See analysis details on SonarCloud

@ghost ghost merged commit 7aa9b4c into main Jan 25, 2024
11 checks passed
@ghost ghost deleted the feature/http-client-performance-fix branch January 25, 2024 12:44
shotexa pushed a commit that referenced this pull request Mar 6, 2024
Signed-off-by: David Poltorak <david.poltorak@iohk.io>
Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
Co-authored-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
Signed-off-by: Shota Jolbordi <shota.jolbordi@iohk.io>
shotexa pushed a commit that referenced this pull request Mar 6, 2024
Signed-off-by: David Poltorak <david.poltorak@iohk.io>
Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
Co-authored-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
Signed-off-by: Shota Jolbordi <shota.jolbordi@iohk.io>
shotexa pushed a commit that referenced this pull request Mar 18, 2024
Signed-off-by: David Poltorak <david.poltorak@iohk.io>
Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
Co-authored-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
Signed-off-by: Shota Jolbordi <shota.jolbordi@iohk.io>
shotexa pushed a commit that referenced this pull request Mar 18, 2024
Signed-off-by: David Poltorak <david.poltorak@iohk.io>
Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
Co-authored-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
shotexa pushed a commit that referenced this pull request Mar 18, 2024
Signed-off-by: David Poltorak <david.poltorak@iohk.io>
Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
Co-authored-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
shotexa pushed a commit that referenced this pull request Mar 18, 2024
Signed-off-by: David Poltorak <david.poltorak@iohk.io>
Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
Co-authored-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
Signed-off-by: Shota Jolbordi <shota.jolbordi@iohk.io>
shotexa pushed a commit that referenced this pull request Mar 18, 2024
Signed-off-by: David Poltorak <david.poltorak@iohk.io>
Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
Co-authored-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
Signed-off-by: Shota Jolbordi <shota.jolbordi@iohk.io>
shotexa pushed a commit that referenced this pull request Mar 18, 2024
Signed-off-by: David Poltorak <david.poltorak@iohk.io>
Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
Co-authored-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
shotexa pushed a commit that referenced this pull request Mar 18, 2024
Signed-off-by: David Poltorak <david.poltorak@iohk.io>
Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
Co-authored-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
Signed-off-by: Shota Jolbordi <shota.jolbordi@iohk.io>
shotexa pushed a commit that referenced this pull request Mar 18, 2024
Signed-off-by: David Poltorak <david.poltorak@iohk.io>
Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
Co-authored-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
Signed-off-by: Shota Jolbordi <shota.jolbordi@iohk.io>
shotexa pushed a commit that referenced this pull request Mar 18, 2024
Signed-off-by: David Poltorak <david.poltorak@iohk.io>
Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
Co-authored-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
Signed-off-by: Shota Jolbordi <shota.jolbordi@iohk.io>
shotexa pushed a commit that referenced this pull request Mar 18, 2024
Signed-off-by: David Poltorak <david.poltorak@iohk.io>
Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
Co-authored-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
Signed-off-by: Shota Jolbordi <shota.jolbordi@iohk.io>
shotexa pushed a commit that referenced this pull request Mar 18, 2024
Signed-off-by: David Poltorak <david.poltorak@iohk.io>
Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
Co-authored-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
Signed-off-by: Shota Jolbordi <shota.jolbordi@iohk.io>
shotexa pushed a commit that referenced this pull request Mar 18, 2024
Signed-off-by: David Poltorak <david.poltorak@iohk.io>
Signed-off-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
Co-authored-by: Benjamin Voiturier <benjamin.voiturier@iohk.io>
This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants