Cannot create new bucket outside of OVA

[hessio]

Problem

When I try to make a new bucket on the OVA outside of the OVA I get an error with the location-constraint. This error does not occur when creating the bucket from inside the OVA.

$ aws s3 mb s3://pictures/ --endpoint-url http://192.168.1.14:31949
make_bucket failed: s3://pictures/ An error occurred (InvalidLocationConstraint) when calling the CreateBucket operation: The specified location-constraint is not valid

Expected behavior

Should be able to create new bucket from a remote machine using the AWS cli (or any s3 client).

How to reproduce

Setup OVA PI-7 and try to create a new bucket from a remote machine. To do this I used VMWare Workstation to run the OVA file and then on my machine in a terminal I tried to create a new bucket using AWS cli.

Deployment information

VMware Workstation 16 Pro running on a Windows 10 machine.
OVA PI-7

Additional information

This also happened to some users during our most recent hackathon as you can see here:

• https://cortxcommunity.slack.com/archives/C01F5NS2820/p1656000845450729?thread_ts=1655988098.412139&cid=C01F5NS2820
• This user also faced the issue but they were only able to list buckets all other s3 commands would fail outside of the OVA https://cortxcommunity.slack.com/archives/C01F5NS2820/p1656012059913989?thread_ts=1655988098.412139&cid=C01F5NS2820

[cortx-admin]

For the convenience of the Seagate development team, this issue has been mirrored in a private Seagate Jira Server: https://jts.seagate.com/browse/CORTX-33841. Note that community members will not be able to access that Jira server but that is not a problem since all activity in that Jira mirror will be copied into this GitHub issue.

[hessio]

I can list buckets or upload data to bucket without any issues but creating a new bucket causes an issue with location constraint

[mukul-seagate11]

@rshenoy0831, did you come across this scenario in your test case?

[rshenoy0831]

we have not came across this behaviour in our testing.
All our testing happens from separate client with OVA Server on different machine.

[stale]

This issue/pull request has been marked as needs attention as it has been left pending without new activity for 4 days. Tagging @mukul-seagate11 for appropriate assignment. Sorry for the delay & Thank you for contributing to CORTX. We will get back to you as soon as possible.

[hessio]

@rshenoy0831 do you have any idea why this might happen for users?

[mukul-seagate11]

@rshenoy0831, can you check the test validation report if this test case was covered?

[rshenoy0831]

@mukul-seagate11 Yes, we have a coverage.
refer test plan - https://jts.seagate.com/browse/TEST-43121
and test execution - https://jts.seagate.com/browse/TEST-43125

[mukul-seagate11]

@hessio, as per QA, it was covered in test plan & test execution

[hessio]

I am going through this testing guide to see but I am finding it difficult to identify what command you are running to create the bucket. It seems to be that when you create a bucket using the command: aws s3api create-bucket --bucket my-bucket --region us-east-1 it works but when you run aws s3 mb s3://mybucket it does not work so maybe you are only using aws s3api create-bucket to test and this is why you never see a failure? Can you confirm @rshenoy0831 ?

[rshenoy0831]

@hessio - we have validated following github page.
https://github.com/Seagate/cortx/blob/main/doc/ova/2.0.0/PI-7/S3_IO_Operations.md
it has aws s3 mb s3://mybucket command anf it worked.

[rshenoy0831]

can we check whether the issue is reproducible if we specify location as default during aws configuration

[hessio]

Yes it is - when you try location as 'None',  'eu-west-1' or 'us-west-1' issue still occurs

[hessio]

@hessio - we have validated following github page. https://github.com/Seagate/cortx/blob/main/doc/ova/2.0.0/PI-7/S3_IO_Operations.md it has aws s3 mb s3://mybucket command anf it worked.

Ok thanks for letting me know. The problem with this issue that it seems to mostly only occur after using the OVA for a few days. Sometimes when you setup the OVA it doesn't occur and other times it does.

[hessio]

ova issues which are reported by community https://github.com/Seagate/cortx/issues?q=is%3Aopen+is%3Aissue+label%3Aova will be further raised to QA and QA will try to verify these issues in next sprint.

[mukul-seagate11]

Haven't observed this issue as per #1644

[hessio]

As you can see this issue still appears with new OVA:

930816@SPR-U930816L001 MINGW64 /
$ aws s3 mb s3://test --endpoint-url http://192.168.1.13:30080
make_bucket failed: s3://test An error occurred (InvalidLocationConstraint) when calling the CreateBucket operation: The specified location-constraint is not valid

930816@SPR-U930816L001 MINGW64 /
$ ssh root@192.168.1.13
root@192.168.1.13's password:
Last login: Tue Sep 13 11:49:19 2022 from 192.168.1.19
[root@cortx-ova ~]# aws s3 mb s3://test
make_bucket: test
[root@cortx-ova ~]#

[hessio]

@mukul-seagate11 this issue is only faced if you try to create a new bucket from a remote machine. When you try to create a bucket on the OVA terminal there is no issue. So in this #1644 you have not tested for this scenario. And as you can see from above comment that when I tested this issue it failed the test.

[mukul-seagate11]

@hessio, this is not in the requirement so only 1 S3 instance is defined

[hessio]

Yes I only have 1 s3 instance. The remote machine is just the host machine that I have my VMWare client installed on

[mukul-seagate11]

@hessio, Ok but with remote machine, the IO functionality hasn't tested yet

[hessio]

Ok ya that's fine, would we be able to add his functionality to the testing?

[mukul-seagate11]

Ok ya that's fine, would we be able to add his functionality to the testing?

Not yet as its not in the roadmap yet

[hessio]

Ok when is the roadmap being updated? I think this is an important feature for using the OVA