Discussion py27-aws-requests-auth or py27-boto into the OPNsense ports tree?

[ndejong]

This github-issue request follows on from #88 and originates from this code review opnsense/plugins#777 (comment)

The code review has recommended that the Python modules being used in the plugin should ideally come through via a BSD ports package source rather than being directly included as code under the plugin distribution itself - use of the requestsaws module is highlighted in this regard.

Unfortunately the requestsaws module is not available and I'm not to fussed about it, however there is an alternative that appears to be much better which I'm happy to migrate to py27-aws-requests-auth

What I would flag however is that the boto library is the perhaps more common "go to" library when working with AWS services and thus might be a more appropriate choice in the long run.

Have reached a small stalling point with the ConfigSync Plugin until we know which is more appropriate, happy to be patient and happy to assist to figure it out if required.

[ndejong]

Actually, with-respect-to boto I believe I've named the older boto package, it should be py27-boto3

• https://boto3.readthedocs.io/en/latest/
• https://www.freebsd.org/cgi/ports.cgi?query=py27-boto&stype=all&sektion=all

The concern to my mind is that each of these options (py27-aws-requests-auth and py27-boto3) creates a fairly long chain of dependent packages which expands the code footprint and in turn may lead to new problems - not sure how I feel about all that other code coming along for the ride I guess...

[AdSchellevis]

@ndejong I think I would go for py27-aws-requests-auth, the api looks quite decent. I can try to build the package for you if you like.

[ndejong]

Something that I've overlooked with py27-aws-requests-auth approach is that the AWS v4 signing scheme seems to require additional information about which region to send requests to where as previous schemes did not.

The Boto3 approach does not seem to need the region and yet they are using v4 signing so there must be some kind of magic that needs to be performed to discover an S3 bucket region - there is some indication in the AWS docs that says you should just use us-east-1 for "services with a globally unique endpoint" however I don't believe S3 meets that definition.

@AdSchellevis - very happy to receive assistance on this, I'll certainly experiment with this further myself too.

[AdSchellevis]

@ndejong I don't have time to digg into the connectivity options, but here are the packages:

pkg add -f https://nightly.opnsense.org/py27-aws-requests-auth-0.4.0.txz 
pkg add -f https://nightly.opnsense.org/py27-xmltodict-0.11.0.txz

It doesn't look like this adds a lot of build time, so if it helps, I can add both to out standard build list.

[ndejong]

Most excellent

• the py27-xmltodict package works nicely, if you are happy with it then do commit it to the build
• playing with the py27-aws-requests-auth package now to address the region quirk as described above

[ndejong]

Unfortunately there are several issues with py27-aws-requests-auth

Issue 1

• missing x-amz-content-sha256 header required for S3 requests that use v4 signatures
• this issue has very recently been addressed in the underlying source but has not yet reached the package distribution
• DavidMuller/aws-requests-auth@e5926a2#diff-96c4e6f4e2ef07f0726a5fa460fe8bdf

$ ./StorageInterfaceAws.py sync_current
{"status": "fail", "timestamp": 1534641520.721913, "message": "Unable to PUT object", "data": "Missing required header for this request: x-amz-content-sha256"}

Issue 2

• the module does not handle or include the user supplied x-amz-meta- headers into the signing mechanism which prevents the request from being accepted by AWS S3.
• after hacking my way around issue 1 above by computing and injecting the x-amz-content-sha256 value a new problem turned up with meta headers

<Error>
<Code>AccessDenied</Code>
<Message>There were headers present in the request which were not signed</Message>
<HeadersNotSigned>x-amz-meta-md5, x-amz-meta-bytes, x-amz-meta-hostid, content-md5, x-amz-meta-filetype, x-amz-meta-mtime</HeadersNotSigned>
<RequestId>XXXXXXXXXXXX</RequestId>
<HostId>XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX</HostId>
</Error>

• if you look at the code for signed_headers there is no attempt to include any user assigned x-amz-meta- headers
• I'm making use of x-amz-meta- to mark the objects with meta data to help identify them

Few different options I guess:-

• accept the manual in-plugin import of the requests-aws code
• consider adopting boto3
• work with the upstream aws-requests-auth to address the issues with their library.

My sense is that boto3 might be worth it, by the look of things it does support other S3 compatible storage mechanisms:-

• https://developers.digitalocean.com/documentation/spaces/
• https://gist.github.com/gleicon/2b8acb9f9c0f22753eaac227ff997b34

[AdSchellevis]

@ndejong I guess you're right, I've build a package for you, although I'm not sure I do have all the dependencies straight.

Can you try:
pkg add -f https://nightly.opnsense.org/py27-boto3-1.4.0.txz

[ndejong]

Thanks @AdSchellevis - I've just checked https://nightly.opnsense.org/py27-boto3-1.4.0.txz and it appears there are missing dependencies

root@OPNsense:/tmp # pkg add -f https://nightly.opnsense.org/py27-boto3-1.4.0.txz
Fetching py27-boto3-1.4.0.txz: 100%   99 KiB  25.3kB/s    00:04    
Installing py27-boto3-1.4.0...
pkg: Missing dependency 'py27-botocore'

Here is another approach that I thought about yesterday. The module originally in use python-requests-aws is available as a pip module but not as a *BSD package module - is it worth simply pkg packaging this thing?

On the plus side we get to avoid a long chain of new package dependencies on the downside we will inherit the responsibility of a new package and the AWS request signing mechanism it uses is not the latest v4, though it seems unlikely this AWS scheme will go away anytime soon.

I've not dealt with packaging a third-party items like this before, happy to take it on if the resulting package is something you'd accept.

Let me know your thoughts

[AdSchellevis]

@ndejong it could be worth packaging it, but ideally this should be done upstream (FreeBSD ports). I'm not sure what the maintenance status is of the package, it seems to be without activity for the last 3 years.

The boto3 package is already in there, which is practical, I could probably deliver you the list of dependencies later this week. The other small thing is that boto3 seems to be actively maintained, but to be honest, I haven't used both of them so I don't have any real preference either in this regard.

[ndejong]

Okay - thanks for taking the time to consider the options here - I I'll wait for the boto3 packages, perhaps the availability of boto3 will give rise to other interesting AWS functionality at a later time too.

[AdSchellevis]

@ndejong here you are https://nightly.opnsense.org/20180821/All/ this should contain all dependencies.

[ndejong]

Thanks - confirming that adding the packages in the following order on a new 18.7.1 instance does not throw any errors:-

pkg add -f https://nightly.opnsense.org/20180821/All/py27-docutils-0.14_3.txz
pkg add -f https://nightly.opnsense.org/20180821/All/py27-jmespath-0.9.3.txz
pkg add -f https://nightly.opnsense.org/20180821/All/py27-dateutil-2.7.3.txz
pkg add -f https://nightly.opnsense.org/20180821/All/py27-futures-3.2.0.txz
pkg add -f https://nightly.opnsense.org/20180821/All/py27-botocore-1.10.65.txz
pkg add -f https://nightly.opnsense.org/20180821/All/py27-s3transfer-0.1.13.txz
pkg add -f https://nightly.opnsense.org/20180821/All/py27-boto3-1.4.0.txz

However - I'm yet to implement usage of the boto3 libraries into the Configuration Sync plugin etc, will report back once I've done so

[AdSchellevis]

@ndejong thanks for confirming, when available in the standard repository the dependencies will be handled automatically.