fix swift: fixed missing Content-Type headers issue #2665

FUSAKLA · 2020-05-26T22:32:01Z

Signed-off-by: Martin Chodur m.chodur@seznam.cz
Fixes #713

@sudhi-vm PTAL as a maintainer of the Swift storage integration

I added CHANGELOG entry for this change.
Change is not relevant to the end user.

Changes

Enabling the Full option in ListOpts. This should provide missing headers for the gophercloud library to be able to determine content-type of the response.
Fix GetRange which did not handle the length: -1 correctly as not set but used it leading to incorrect range bytes=1--1 instead of bytes=1-. See the Range parameter of the GET method in the Swift API docs.

Verification

Before, two acceptance e2e tests for object storage were failing

=== RUN   TestObjStore_AcceptanceTest_e2e
=== PAUSE TestObjStore_AcceptanceTest_e2e
=== CONT  TestObjStore_AcceptanceTest_e2e
--- FAIL: TestObjStore_AcceptanceTest_e2e (0.94s)
=== RUN   TestObjStore_AcceptanceTest_e2e/inmem
    --- PASS: TestObjStore_AcceptanceTest_e2e/inmem (0.00s)
=== RUN   TestObjStore_AcceptanceTest_e2e/filesystem
=== PAUSE TestObjStore_AcceptanceTest_e2e/filesystem
    TestObjStore_AcceptanceTest_e2e: foreach.go:32: GCS found in THANOS_TEST_OBJSTORE_SKIP array. Skipping.
    TestObjStore_AcceptanceTest_e2e: foreach.go:32: S3 found in THANOS_TEST_OBJSTORE_SKIP array. Skipping.
    TestObjStore_AcceptanceTest_e2e: foreach.go:32: AZURE found in THANOS_TEST_OBJSTORE_SKIP array. Skipping.
=== CONT  TestObjStore_AcceptanceTest_e2e/filesystem
    --- PASS: TestObjStore_AcceptanceTest_e2e/filesystem (0.01s)
=== RUN   TestObjStore_AcceptanceTest_e2e/swift
    TestObjStore_AcceptanceTest_e2e/swift: swift.go:316: created temporary container for swift tests with name test-65c69e700dd980c8-testobjstore-acceptancetest-e2e-swift
=== PAUSE TestObjStore_AcceptanceTest_e2e/swift
    TestObjStore_AcceptanceTest_e2e: foreach.go:32: COS found in THANOS_TEST_OBJSTORE_SKIP array. Skipping.
    TestObjStore_AcceptanceTest_e2e: foreach.go:32: ALIYUNOSS found in THANOS_TEST_OBJSTORE_SKIP array. Skipping.
=== CONT  TestObjStore_AcceptanceTest_e2e/swift
    TestObjStore_AcceptanceTest_e2e/swift: testing.go:130: testing.go:130:
        
        	exp: "test-data@"
        
        	got: "@test-data@"
        
        Diff:
        --- Expected
        +++ Actual
        @@ -1 +1 @@
        -test-data@
        +@test-data@
        
        
    TestObjStore_AcceptanceTest_e2e/swift: testing.go:58: iterating over bucket objects failed: Cannot extract names from response with content-type: []
    TestObjStore_AcceptanceTest_e2e/swift: swift.go:321: deleting container test-65c69e700dd980c8-testobjstore-acceptancetest-e2e-swift failed: Expected HTTP response code [] when accessing [DELETE https://test-swift-proxy.test.dszn.cz/v1/AUTH_ddbe05efeb1142508d781a92d63a781e/test-65c69e700dd980c8-testobjstore-acceptancetest-e2e-swift], but got 409 instead
        <html><h1>Conflict</h1><p>There was a conflict when trying to complete your request.</p></html>
    --- FAIL: TestObjStore_AcceptanceTest_e2e/swift (2.01s)
FAIL

Process finished with exit code 1

Now all tests are passing

=== RUN   TestObjStore_AcceptanceTest_e2e
=== PAUSE TestObjStore_AcceptanceTest_e2e
=== CONT  TestObjStore_AcceptanceTest_e2e
--- PASS: TestObjStore_AcceptanceTest_e2e (0.91s)
=== RUN   TestObjStore_AcceptanceTest_e2e/inmem
    --- PASS: TestObjStore_AcceptanceTest_e2e/inmem (0.00s)
=== RUN   TestObjStore_AcceptanceTest_e2e/filesystem
=== PAUSE TestObjStore_AcceptanceTest_e2e/filesystem
    TestObjStore_AcceptanceTest_e2e: foreach.go:32: GCS found in THANOS_TEST_OBJSTORE_SKIP array. Skipping.
    TestObjStore_AcceptanceTest_e2e: foreach.go:32: S3 found in THANOS_TEST_OBJSTORE_SKIP array. Skipping.
    TestObjStore_AcceptanceTest_e2e: foreach.go:32: AZURE found in THANOS_TEST_OBJSTORE_SKIP array. Skipping.
=== CONT  TestObjStore_AcceptanceTest_e2e/filesystem
    --- PASS: TestObjStore_AcceptanceTest_e2e/filesystem (0.00s)
=== RUN   TestObjStore_AcceptanceTest_e2e/swift
    TestObjStore_AcceptanceTest_e2e/swift: swift.go:324: created temporary container for swift tests with name test-281cd3eb693829b6-testobjstore-acceptancetest-e2e-swift
=== PAUSE TestObjStore_AcceptanceTest_e2e/swift
    TestObjStore_AcceptanceTest_e2e: foreach.go:32: COS found in THANOS_TEST_OBJSTORE_SKIP array. Skipping.
    TestObjStore_AcceptanceTest_e2e: foreach.go:32: ALIYUNOSS found in THANOS_TEST_OBJSTORE_SKIP array. Skipping.
=== CONT  TestObjStore_AcceptanceTest_e2e/swift
    --- PASS: TestObjStore_AcceptanceTest_e2e/swift (4.92s)
PASS

Process finished with exit code 0

To be honest I do not know how this could have worked for anyone else. If it did maybe it could have been cased by different version of Swift for example? Or changes in the Gophercloud library?

I'd be really glad to hear from anyone using Swift already what version of Swift they have and how the Thanos is configured exactly.

Our Thanos objstore config for Swift:

type: SWIFT
config:
  auth_url: "https://foo.bar:5000/v3/"
  username: "thanos"
  user_domain_name: "general_dev"
  password: "*****"
  domain_name: "general_dev"
  project_name: "s_98"
  project_domain_name: "general_dev"
  region_name: "Test"
  container_name: "thanos"

Output of the /info of our Swift cluster

{
  "bulk_delete": {
    "max_failed_deletes": 1000,
    "max_deletes_per_request": 10000
  },
  "container_quotas": {},
  "tempurl": {
    "outgoing_remove_headers": [
      "x-object-meta-*"
    ],
    "incoming_remove_headers": [
      "x-timestamp"
    ],
    "incoming_allow_headers": [],
    "methods": [
      "GET",
      "HEAD",
      "PUT",
      "POST",
      "DELETE"
    ],
    "outgoing_allow_headers": [
      "x-object-meta-public-*"
    ]
  },
  "bulk_upload": {
    "max_failed_extractions": 1000,
    "max_containers_per_extraction": 10000
  },
  "container_sync": {
    "realms": {}
  },
  "ratelimit": {
    "account_ratelimit": 10,
    "max_sleep_time_seconds": 10,
    "container_ratelimits": [
      [
        0,
        20
      ]
    ],
    "container_listing_ratelimits": [
      [
        0,
        200
      ]
    ]
  },
  "slo": {
    "max_manifest_segments": 1000,
    "min_segment_size": 1,
    "max_manifest_size": 2097152
  },
  "account_quotas": {},
  "swift": {
    "max_file_size": 1073741826,
    "account_listing_limit": 10000,
    "account_autocreate": true,
    "max_meta_count": 90,
    "max_meta_value_length": 256,
    "container_listing_limit": 10000,
    "extra_header_count": 0,
    "max_meta_overall_size": 4096,
    "version": "2.15.1",
    "max_meta_name_length": 128,
    "max_header_size": 8192,
    "policies": [
      {
        "name": "Policy-0",
        "aliases": "Policy-0"
      },
      {
        "default": true,
        "name": "dev-test",
        "aliases": "dev-test"
      }
    ],
    "max_object_name_length": 1024,
    "max_account_name_length": 256,
    "strict_cors_mode": true,
    "allow_account_management": true,
    "max_container_name_length": 256
  },
  "swift3": {
    "max_upload_part_num": 1000,
    "allow_multipart_uploads": true,
    "max_bucket_listing": 1000,
    "version": "1.11.1.dev142",
    "max_parts_listing": 1000,
    "max_multi_delete_objects": 1000
  }
}

FUSAKLA · 2020-05-26T22:42:56Z

Linting failed due to OOM kill, but I do not have the permission to rerun it :/

Signed-off-by: Martin Chodur <m.chodur@seznam.cz>

GiedriusS

Sorry for the late response. AFAICT the changes look good from the code's perspective. I wonder how did this work before. I will try to test this out practically in a few days.

GiedriusS · 2020-07-01T20:36:52Z

Tested this with https://github.com/beaukode/docker-swift-onlyone-authv2-keystone. What has tripped me is that the default region is called RegionOne. I have used the following exports:

OS_AUTH_URL=http://127.0.0.1:5000/v2.0
OS_PASSWORD=s3cr3t
OS_PROJECT_NAME=admin
OS_REGION_NAME=RegionOne
OS_USERNAME=admin
THANOS_TEST_OBJSTORE_SKIP=GCS,S3,AZURE,COS,ALIYUNOSS

Not sure how no one has noticed this yet. Thanks a lot.

…nt-type Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com>

GiedriusS · 2020-07-01T20:43:57Z

I have fixed up the CHANGELOG.md file so that it could be merged. At first, I have done it via the Web UI but I forgot to add a DCO there so I have done that via the command-line now. I hope you don't mind. Merging on green!

GiedriusS · 2020-07-01T20:59:13Z

I guess I'll merge this to fix the current code as-is and then let's work on #2732

bwplotka

👍

FUSAKLA · 2020-07-02T09:41:02Z

Cool thanks 👍
When I realized there are bigger issues with the Swift integration I kind of abandoned this and switched to the refactor of the Swift support.

FUSAKLA mentioned this pull request May 26, 2020

Swift upload: "new bucket block: list chunk files: Cannot extract names from response with content-type: []" #713

Closed

FUSAKLA mentioned this pull request May 28, 2020

feat tools: bucket replicate allows to filter multiple compaction and… #2671

Merged

2 tasks

FUSAKLA added 4 commits May 28, 2020 12:05

fix swift: fixed missing Content-Type headers issue

9c15cfb

Signed-off-by: Martin Chodur <m.chodur@seznam.cz>

chore CHANGELOG: add PR id

bbfbd84

Signed-off-by: Martin Chodur <m.chodur@seznam.cz>

fix swift: fixed invalid lower range boundary

b5201cb

Signed-off-by: Martin Chodur <m.chodur@seznam.cz>

fix swift: fixed typo in ENV variable for testing bucket

d68542f

Signed-off-by: Martin Chodur <m.chodur@seznam.cz>

FUSAKLA force-pushed the fus-swift-fix-content-type branch from 34a33fb to d68542f Compare May 28, 2020 10:11

FUSAKLA mentioned this pull request Jun 8, 2020

objstore/swift: use ncw/swift and support large files for OpenStack Swift #2732

Merged

2 tasks

GiedriusS approved these changes Jun 25, 2020

View reviewed changes

Merge remote-tracking branch 'origin/master' into fus-swift-fix-conte…

1f4db39

…nt-type Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com>

GiedriusS force-pushed the fus-swift-fix-content-type branch from 437e4eb to 1f4db39 Compare July 1, 2020 20:42

GiedriusS mentioned this pull request Jul 1, 2020

Add Swift integration tests #2829

Closed

GiedriusS merged commit 5f06821 into thanos-io:master Jul 1, 2020

bwplotka reviewed Jul 1, 2020

View reviewed changes

cuotos mentioned this pull request Jul 7, 2020

Query: correctly marshal errors to JSON and ignore if nil #2848

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix swift: fixed missing Content-Type headers issue #2665

fix swift: fixed missing Content-Type headers issue #2665

FUSAKLA commented May 26, 2020

FUSAKLA commented May 26, 2020

GiedriusS left a comment •

edited

GiedriusS commented Jul 1, 2020

GiedriusS commented Jul 1, 2020

GiedriusS commented Jul 1, 2020

bwplotka left a comment

FUSAKLA commented Jul 2, 2020

fix swift: fixed missing Content-Type headers issue #2665

fix swift: fixed missing Content-Type headers issue #2665

Conversation

FUSAKLA commented May 26, 2020

Changes

Verification

FUSAKLA commented May 26, 2020

GiedriusS left a comment • edited

Choose a reason for hiding this comment

GiedriusS commented Jul 1, 2020

GiedriusS commented Jul 1, 2020

GiedriusS commented Jul 1, 2020

bwplotka left a comment

Choose a reason for hiding this comment

FUSAKLA commented Jul 2, 2020

GiedriusS left a comment •

edited