Running tests on minio #358

fao89 · 2020-02-21T20:41:52Z

https://pulp.plan.io/issues/6155
ref #6155

fao89 · 2020-02-29T15:19:50Z

pulpcore tests are failing
https://travis-ci.org/pulp/pulp_file/jobs/656658172?utm_medium=notification&utm_source=github_status
for now, I'm running only pulp_file test

fao89 · 2020-02-29T15:22:03Z

.travis/before_install.sh

+  # Hacking botocore:
+  sed -i "/gunicorn/a RUN sed -i \"s/hasattr(body, 'read')/getattr(body, '_size', None)/g\" \$(pip show botocore | grep -i location | awk '{ print \$2 }')/botocore/handlers.py" ./containers/images/pulp/Dockerfile.j2
+  cat ./containers/images/pulp/Dockerfile.j2


thank you @daviddavis
your tip led me to open this PR:
boto/botocore#1990

Excellent I've subscribed to the PR and plan on following up next week if no one responds.

Can you add a link to your PR in the comments?

daviddavis · 2020-02-29T16:28:51Z

.travis/script.sh

@@ -89,8 +89,10 @@ export CMD_STDIN_PREFIX="sudo kubectl exec -i $PULP_API_POD --"
 cat unittest_requirements.txt | $CMD_STDIN_PREFIX bash -c "cat > /tmp/test_requirements.txt"
 $CMD_PREFIX pip3 install -r /tmp/test_requirements.txt

-# Run unit tests.
-$CMD_PREFIX bash -c "PULP_DATABASES__default__USER=postgres django-admin test --noinput /usr/local/lib/python${TRAVIS_PYTHON_VERSION}/site-packages/pulp_file/tests/unit/"
+if [[ "$TEST" != 's3' ]]; then


Why are we skipping the unit tests for s3?

daviddavis · 2020-02-29T16:30:07Z

pulp_file/tests/functional/conftest.py

+
+def pytest_runtest_setup(item):
+    """Adding delay."""
+    time.sleep(0.5)


This adds a 0.5 sleep before every test? Is that needed to throttle connections to minio?

If so, I wonder if we should make this a configuration setting so it doesn't impact non-minio test runs?

I forgot to remove that

daviddavis · 2020-02-29T16:30:56Z

pulp_file/tests/functional/utils.py

@@ -173,7 +173,7 @@ def monitor_task(task_href):
    completed = ["completed", "failed", "canceled"]
    task = tasks.read(task_href)
    while task.state not in completed:
-        sleep(2)
+        sleep(1)


Is this just an optimization or is it minio related?

just an optimization, I think 2 seconds is too much

daviddavis · 2020-02-29T16:31:36Z

Overall, great job. I think I like this solution better than the localstack one. What do you think?

fao89 · 2020-02-29T17:42:50Z

Overall, great job. I think I like this solution better than the localstack one. What do you think?

I agree minio is simpler

daviddavis · 2020-03-01T16:57:04Z

.travis/script.sh

@@ -106,6 +113,11 @@ export PYTHONPATH=$TRAVIS_BUILD_DIR:$TRAVIS_BUILD_DIR/../pulpcore:${PYTHONPATH}

 set -u

+if [[ "$TEST" == 's3' ]]; then
+  pytest -v -r sx --color=yes --pyargs pulp_file.tests.functional || show_logs_and_return_non_zero
+  exit


Is this to not run the pulpcore tests? If so, I have a PR to get them passing:

pulp/pulpcore#567

yes, it was because of that, I'm linking your PR as Required PR here

daviddavis · 2020-03-02T12:53:56Z

@mikedep333 @dkliban either of you want to review this before we merge and apply this to the plugin_template?

Required PR: pulp/pulp_file#358 https://pulp.plan.io/issues/6155 closes #6155

Required PR: pulp/pulp_file#358 Required PR: pulp/pulp-operator#35 https://pulp.plan.io/issues/6155 closes #6155

daviddavis · 2020-03-10T15:55:00Z

template_config.yml

+  aws_storage_bucket_name: '"pulp3"'
+  default_file_storage: '"storages.backends.s3boto3.S3Boto3Storage"'
+  media_root: ''
+  s3_use_sigv4: 'true'


I think this needs to be removed?

yep, I forgot to run plugin_template after the last change, I'll do it for all PRs

Required PR: pulp/pulp_file#358 Required PR: pulp/pulp-operator#35 https://pulp.plan.io/issues/6155 closes #6155

Required PR: pulp/pulp-operator#35 Required PR: pulp/pulpcore#568 https://pulp.plan.io/issues/6155 ref #6155

Required PR: pulp/pulp_file#358 Required PR: pulp/pulp-operator#35 https://pulp.plan.io/issues/6155 closes #6155

fao89 force-pushed the 6155 branch 8 times, most recently from f77b79d to 0b953ef Compare February 27, 2020 18:17

fao89 closed this Feb 27, 2020

fao89 reopened this Feb 29, 2020

fao89 force-pushed the 6155 branch 7 times, most recently from 9ca9765 to 109315c Compare February 29, 2020 14:12

fao89 marked this pull request as ready for review February 29, 2020 15:16

fao89 requested a review from a team February 29, 2020 15:16

fao89 commented Feb 29, 2020

View reviewed changes

daviddavis reviewed Feb 29, 2020

View reviewed changes

fao89 force-pushed the 6155 branch from 109315c to 525f095 Compare February 29, 2020 17:41

fao89 force-pushed the 6155 branch from 525f095 to 753d809 Compare February 29, 2020 17:44

fao89 mentioned this pull request Feb 29, 2020

Running tests on localstack #362

Closed

fao89 force-pushed the 6155 branch from 753d809 to a1176e8 Compare February 29, 2020 17:56

fao89 force-pushed the 6155 branch from a1176e8 to 85b43fe Compare February 29, 2020 18:55

daviddavis reviewed Mar 1, 2020

View reviewed changes

fao89 force-pushed the 6155 branch 2 times, most recently from 4ee1ecb to 0ffe3db Compare March 1, 2020 21:49

fao89 added a commit to fao89/pulpcore that referenced this pull request Mar 4, 2020

Running tests on minio

5633480

Required PR: pulp/pulp_file#358 https://pulp.plan.io/issues/6155 closes #6155

fao89 added a commit to fao89/pulpcore that referenced this pull request Mar 4, 2020

Running tests on minio

535fece

Required PR: pulp/pulp_file#358 https://pulp.plan.io/issues/6155 closes #6155

fao89 added a commit to fao89/pulpcore that referenced this pull request Mar 5, 2020

Running tests on minio

c4095e8

Required PR: pulp/pulp_file#358 https://pulp.plan.io/issues/6155 closes #6155

fao89 added a commit to fao89/pulpcore that referenced this pull request Mar 5, 2020

Running tests on minio

a9062d7

Required PR: pulp/pulp_file#358 https://pulp.plan.io/issues/6155 closes #6155

fao89 force-pushed the 6155 branch from 0ffe3db to b7c47b1 Compare March 5, 2020 02:15

fao89 added a commit to fao89/pulpcore that referenced this pull request Mar 5, 2020

Running tests on minio

86d616e

Required PR: pulp/pulp_file#358 https://pulp.plan.io/issues/6155 closes #6155

fao89 added a commit to fao89/pulpcore that referenced this pull request Mar 5, 2020

Running tests on minio

9f48603

Required PR: pulp/pulp_file#358 https://pulp.plan.io/issues/6155 closes #6155

fao89 added a commit to fao89/pulpcore that referenced this pull request Mar 5, 2020

Running tests on minio

e8a6ed7

Required PR: pulp/pulp_file#358 https://pulp.plan.io/issues/6155 closes #6155

fao89 added a commit to fao89/pulpcore that referenced this pull request Mar 6, 2020

Running tests on minio

df99514

Required PR: pulp/pulp_file#358 Required PR: pulp/pulp-operator#35 https://pulp.plan.io/issues/6155 closes #6155

fao89 force-pushed the 6155 branch from b7c47b1 to 0376806 Compare March 6, 2020 14:23

daviddavis reviewed Mar 10, 2020

View reviewed changes

fao89 added a commit to fao89/pulpcore that referenced this pull request Mar 10, 2020

Running tests on minio

a67a9fc

Required PR: pulp/pulp_file#358 Required PR: pulp/pulp-operator#35 https://pulp.plan.io/issues/6155 closes #6155

Running tests on minio

bbc46e1

Required PR: pulp/pulp-operator#35 Required PR: pulp/pulpcore#568 https://pulp.plan.io/issues/6155 ref #6155

fao89 force-pushed the 6155 branch from 0376806 to bbc46e1 Compare March 10, 2020 16:12

daviddavis approved these changes Mar 10, 2020

View reviewed changes

fao89 added a commit to fao89/pulpcore that referenced this pull request Mar 10, 2020

Running tests on minio

bca9bf4

Required PR: pulp/pulp_file#358 Required PR: pulp/pulp-operator#35 https://pulp.plan.io/issues/6155 closes #6155

daviddavis approved these changes Mar 10, 2020

View reviewed changes

daviddavis merged commit 5b97616 into pulp:master Mar 10, 2020

Running tests on minio #358

Running tests on minio #358

fao89 commented Feb 21, 2020

fao89 commented Feb 29, 2020

fao89 Feb 29, 2020

daviddavis Feb 29, 2020

fao89 Feb 29, 2020

daviddavis Feb 29, 2020

daviddavis Feb 29, 2020 •

edited

fao89 Feb 29, 2020

daviddavis Feb 29, 2020

fao89 Feb 29, 2020

daviddavis commented Feb 29, 2020

fao89 commented Feb 29, 2020

daviddavis Mar 1, 2020

fao89 Mar 1, 2020

daviddavis commented Mar 2, 2020

daviddavis Mar 10, 2020

fao89 Mar 10, 2020

Running tests on minio #358

Running tests on minio #358

Conversation

fao89 commented Feb 21, 2020

fao89 commented Feb 29, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daviddavis Feb 29, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daviddavis commented Feb 29, 2020

fao89 commented Feb 29, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daviddavis commented Mar 2, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daviddavis Feb 29, 2020 •

edited