Adding subdir for Distributed Wrangler Load Tests

[malachi-constant]

Feature

• Adds dedicated directory for load tests for distributed wrangler methods

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[malachi-constant]

Two things.

1. Do we want to keep our load tests in a dedicated subdirectory? I figured adding them to tests/ may end up polluting CI as it runs pytest tests/*

2. @jaidisido I tried putting in a decorator to handle elapsed time, but was having trouble getting pytest to collect the functions when decorated. E.g.

import time
from functools import wraps


def elapsed_time(func):
    @wraps(func)
    def elapsed_time_wrapper(*args, **kwargs):
        start_time = time.perf_counter()
        result = func(*args, **kwargs)
        end_time = time.perf_counter()
        total_time = end_time - start_time
        print(f"Function {func.__name__}{args} {kwargs} Took {total_time:.4f} seconds")
        return result

    return

PytestCollectionWarning: cannot collect 'test_s3_select' because it is not a function. def __call__(self, *args: object, **kwargs: object):

[malachi-constant]

AWS CodeBuild CI Report

• CodeBuild project: GitHubLoadTests5656BB24-s6u9F3qN9oFy
• Commit ID: a4800d6
• Result: SUCCEEDED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[jaidisido]

1. I am in favour of separating load_tests into their own directory like you suggest. @kukushking / @cnfait what do you think?
2. Should we simply use the --durations=0 flag of pytest?