Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add more integration tests #144

Open
3 of 25 tasks
achimnol opened this issue Nov 26, 2018 · 0 comments
Open
3 of 25 tasks

Add more integration tests #144

achimnol opened this issue Nov 26, 2018 · 0 comments
Assignees
Labels
area:infrastructure Infrastructure-related issues comp:client Related to Client component

Comments

@achimnol
Copy link
Member

achimnol commented Nov 26, 2018

First, read the official integration testing guide. Let's realize it.

Some of below functionalities are already covered in the test cases of the manager, but we need to test also API's input validation and matching of the client-side and server-side implementation.

  • Kernel integration tests (tests/test_kernel.py)
    • Use this a starting point and reference. (It also may require some updates though)
  • Admin integration tests (tests/test_admin.py) -> needs updates!
    • Create, delete, modify domains: use testing-XXXX format (with resource limits)
    • Create, delete, modify groups (with resource limits)
    • Create, delete, modify keypairs and users in different domains and groups (with resource limits)
    • Check if the current user's resource limit reflects the domain/group limits as well if configured.
  • Advanced kernel integration tests (tests/test_kernel.py)
    • As a fixture, you need to create a domain, a group, and a user who belongs to them.
    • Execute with custom environment variables using the above user
    • Execute via websockets (stream APIs) using the above user
    • Execute with the batch mode's build/clean commands, including cases with both satisfying and exceeding the batch-mode API's file number/size limits
    • Activate service ports after creating kernels, and check if they give valid responses (e.g., some HTML codes for web-based container services like Jupyter)
    • Group/domain segregation
      • As a fixture, you need to create two or more domain/group/user sets.
      • Check if running kernels for a specific domain and group is not listed in other user in a different domain and group.
    • Resource limits
      • Execute with different resource limits and check/hit those limits (e.g., check the number of CPU cores, check if an OOM event (forced termination) occurs)
      • Try to create kernels whose creation configs exceed the configured domain, group, user (keypair) resource limits and check if the limits are enforced as expected.
  • VFolder integration tests (tests/test_vfolder.py)
    • Create and delete personal vfolders, check them using the listing API
    • Hit the vfolder limit of the keypair resource policy by creating too many vfolders
    • Upload/download small (10 MiB) and large (10 GiB) random-generated files with hash checks
    • Group vfolders
      • As a fixture, you need to create two or more groups and corresponding domain admin users.
      • Create and delete group vfolders, check them using the listing API
      • Check if users who belong to only a specific group cannot list & access another group's vfolders.
    • Domain segregation
      • As a fixture, you need to create two or more domain/group/user sets.
      • Check if vfolders for a specific domain cannot send/receive invitations to/from users in a different domain and group.
@achimnol achimnol pinned this issue Dec 14, 2018
adrysn referenced this issue in lablup/backend.ai-client-py Mar 27, 2019
adrysn referenced this issue in lablup/backend.ai-client-py Mar 28, 2019
* Add image tests.
* Add extra aliases field when querying images.
adrysn referenced this issue in lablup/backend.ai-client-py Mar 28, 2019
adrysn referenced this issue in lablup/backend.ai-client-py Mar 29, 2019
adrysn referenced this issue in lablup/backend.ai-client-py Mar 29, 2019
achimnol referenced this issue in lablup/backend.ai-client-py Jun 29, 2019
* Remove test/test_integration.py as it is now described in #43.
@achimnol achimnol transferred this issue from lablup/backend.ai-client-py Feb 5, 2020
@achimnol achimnol added comp:client Related to Client component area:infrastructure Infrastructure-related issues labels Feb 5, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:infrastructure Infrastructure-related issues comp:client Related to Client component
Projects
None yet
Development

No branches or pull requests

5 participants