Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[report|collector|policies] Add an S3 protocol upload option #3447

Merged
merged 2 commits into from
Jan 19, 2024

Conversation

TrevorBenson
Copy link
Member

@TrevorBenson TrevorBenson commented Dec 16, 2023

Closes #3112


Please place an 'X' inside each '[]' to confirm you adhere to our Contributor Guidelines

  • Is the commit message split over multiple lines and hard-wrapped at 72 characters?
  • Is the subject and message clear and concise?
  • Does the subject start with [plugin_name] if submitting a plugin patch or a [section_name] if part of the core sosreport code?
  • Does the commit contain a Signed-off-by: First Lastname email@example.com?
  • Are any related Issues or existing PRs properly referenced via a Closes (Issue) or Resolved (PR) line?

Copy link

Congratulations! One of the builds has completed. 🍾

You can install the built RPMs by following these steps:

  • sudo yum install -y dnf-plugins-core on RHEL 8
  • sudo dnf install -y dnf-plugins-core on Fedora
  • dnf copr enable packit/sosreport-sos-3447
  • And now you can install the packages.

Please note that the RPMs should be used only in a testing environment.

@TrevorBenson
Copy link
Member Author

@TurboTurtle The x86_64 RPM build succeeds, however, I don't seem to be able to access repo metadata for installation and testing:

$ sudo dnf install -y sos-4.6.0-1.20231216003429832355.pr3447.82.g9e05d8c7.fc40.noarch
Copr repo for sosreport-sos-3447 owned by packit                                                                                                                                                              454  B/s | 341  B     00:00    
Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:packit:sosreport-sos-3447':
  - Status code: 404 for https://download.copr.fedorainfracloud.org/results/packit/sosreport-sos-3447/fedora-39-x86_64/repodata/repomd.xml (IP: 2600:9000:25f2:ca00:4:bbc1:1840:93a1)
Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:packit:sosreport-sos-3447': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
Ignoring repositories: copr:copr.fedorainfracloud.org:packit:sosreport-sos-3447
Last metadata expiration check: 2:22:12 ago on Fri 15 Dec 2023 02:21:32 PM PST.
No match for argument: sos-4.6.0-1.20231216003429832355.pr3447.82.g9e05d8c7.fc40.noarch
Error: Unable to find a match: sos-4.6.0-1.20231216003429832355.pr3447.82.g9e05d8c7.fc40.noarch

pmoravec added a commit to pmoravec/sos that referenced this pull request Dec 16, 2023
As Fedora 39 is out, let bump its versions in copr builds and GCE
testing.

Sadly, GCE images offer F39 beta only, so far..

Resolves: sosreport#3448
Relevant: sosreport#3447

Signed-off-by: Pavel Moravec <pmoravec@redhat.com>
@pmoravec
Copy link
Contributor

The copr build is not there since we haven't bumped Fedora versions since F39 release: we ask for such builds for F37, F38 and rawhide now.

I opened #3448 to fix that.

@pmoravec
Copy link
Contributor

NACK to the plain extension of install requirements. We should keep the hard requirements as small as possible, to allow installation of sos to as wide range of systems as possible.

E.g. here, python3-boto3 is not in RHEL (8 or 9), just in EPEL. So this change would prevent installing sos on RHEL :)

Rather make the requirement and import conditional, like:

https://github.com/sosreport/sos/blob/main/sos/policies/distros/__init__.py#L29-L33

@arif-ali
Copy link
Member

arif-ali commented Dec 16, 2023

The deb build is failing, as it does some unit tests as part of the build (That's how deb packaging works).

So, we will need to add python3-boto3, so that in debian/control now has the following change

Depends: ${python3:Depends}, ${misc:Depends}, python3-pexpect, python3-magic, python3-packaging, python3-boto3

For the snap (as that will then fail too), can you add boto3 under python-packages in snap/snapcraft.yaml

@dnegreira FYI, does that sound reasonable for the control file, or do we want to avoid doing a Depends on a universe package, see below?

❱❱❱ rmadison python3-boto3
 python3-boto3 | 1.2.2-2         | xenial/universe | all
 python3-boto3 | 1.4.2-1         | bionic/universe | all
 python3-boto3 | 1.9.253-1       | focal/universe  | all
 python3-boto3 | 1.20.34+dfsg-1  | jammy/universe  | all
 python3-boto3 | 1.26.27+dfsg-1  | lunar/universe  | all
 python3-boto3 | 1.26.155+dfsg-1 | mantic/universe | all
 python3-boto3 | 1.28.49+dfsg-1  | noble/universe  | all

@TrevorBenson
Copy link
Member Author

@arif-ali Would adding a Recommends and placing it there be more appropriate in case Universe is not enabled?

@TrevorBenson
Copy link
Member Author

The copr build is not there since we haven't bumped Fedora versions since F39 release: we ask for such builds for F37, F38 and rawhide now.

I opened #3448 to fix that.

For now, I'll spin up a new F38 VM until the F39 issues are resolved.

@arif-ali
Copy link
Member

@arif-ali Would adding a Recommends and placing it there be more appropriate in case Universe is not enabled?

could be an option, but want a +1 from my other fellow colleagues who take on the Ubuntu packaging. But like @pmoravec mentioned, we ought to have a condition of the boto3 being available. Considering that boto3 is in universe in Ubuntu and only in EPEL for EL 8 and EL9.

From the Ubuntu perspective, we expect this to be available for 20.04, 22.04, 23.04, 23.10 and the upcoming 24.04. All of these have boto3 in universe

But happy on the snap side, as that would be the only way to enable it, and it builds on core22 which is a 22.04 base and will ship as standard, and we ship all the libs as required and comes via pip3.

@TrevorBenson
Copy link
Member Author

The copr build is not there since we haven't bumped Fedora versions since F39 release: we ask for such builds for F37, F38 and rawhide now.
I opened #3448 to fix that.

For now, I'll spin up a new F38 VM until the F39 issues are resolved.

@pmoravec The instructions from packit-as-a-service don't seem to work as described for F38 either:

$ sudo dnf install -y dnf-plugins-core
Fedora 38 - x86_64                              3.7 MB/s |  83 MB     00:22    
Fedora 38 openh264 (From Cisco) - x86_64        1.0 kB/s | 2.5 kB     00:02    
Fedora Modular 38 - x86_64                      777 kB/s | 2.8 MB     00:03    
Fedora 38 - x86_64 - Updates                    2.7 MB/s |  36 MB     00:13    
Fedora Modular 38 - x86_64 - Updates            463 kB/s | 2.1 MB     00:04    
Package dnf-plugins-core-4.4.0-1.fc38.noarch is already installed.
Dependencies resolved.
Nothing to do.
Complete!

$ sudo dnf copr enable packit/sosreport-sos-3447
Enabling a Copr repository. Please note that this repository is not part
of the main distribution, and quality may vary.

The Fedora Project does not exercise any power over the contents of
this repository beyond the rules outlined in the Copr FAQ at
<https://docs.pagure.org/copr.copr/user_documentation.html#what-i-can-build-in-copr>,
and packages are not held to any quality or security level.

Please do not file bug reports about these packages in Fedora
Bugzilla. In case of problems, contact the owner of this repository.

Do you really want to enable copr.fedorainfracloud.org/packit/sosreport-sos-3447? [y/N]: y
Error: It wasn't possible to enable this project.
Repository 'fedora-38-x86_64' does not exist in project 'packit/sosreport-sos-3447'.
Available repositories: 'fedora-rawhide-aarch64', 'fedora-rawhide-ppc64le', 'fedora-rawhide-x86_64', 'fedora-rawhide-s390x'

If you want to enable a non-default repository, use the following command:
  'dnf copr enable packit/sosreport-sos-3447 <repository>'
But note that the installed repo file will likely need a manual modification.

Since it was identical behavior to F39 I enabled fedora-rawhide-x86_64 as the only x86_64 based repo suggested. I haven't used copr from packit in at least a year, should it automatically find the fedora-38-x86_64, or should rawhide have the repository metadata for F38?

$ sudo dnf copr enable packit/sosreport-sos-3447 fedora-rawhide-x86_64
Enabling a Copr repository. Please note that this repository is not part
of the main distribution, and quality may vary.

The Fedora Project does not exercise any power over the contents of
this repository beyond the rules outlined in the Copr FAQ at
<https://docs.pagure.org/copr.copr/user_documentation.html#what-i-can-build-in-copr>,
and packages are not held to any quality or security level.

Please do not file bug reports about these packages in Fedora
Bugzilla. In case of problems, contact the owner of this repository.

Do you really want to enable copr.fedorainfracloud.org/packit/sosreport-sos-3447? [y/N]: y
Repository successfully enabled.

$ dnf list sos --showduplicates
Copr repo for sosreport-sos-3447 owned by packit                                                           187  B/s | 341  B     00:01    
Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:packit:sosreport-sos-3447':
  - Status code: 404 for https://download.copr.fedorainfracloud.org/results/packit/sosreport-sos-3447/fedora-38-x86_64/repodata/repomd.xml (IP: 108.138.85.40)
Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:packit:sosreport-sos-3447': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
Fedora Modular 38 - x86_64 - Updates                                                                       417 kB/s | 2.1 MB     00:05    
Ignoring repositories: copr:copr.fedorainfracloud.org:packit:sosreport-sos-3447
Last metadata expiration check: 0:00:01 ago on Sat 16 Dec 2023 09:21:59 PM EST.
Installed Packages
sos.noarch                                                      4.5.1-1.fc38                                                      @anaconda
Available Packages
sos.noarch                                                      4.5.1-1.fc38                                                      fedora   

@pmoravec
Copy link
Contributor

pmoravec commented Dec 17, 2023

I think we need to change https://github.com/sosreport/sos/blob/main/.packit.yaml#L17-L21 to target fedora-all.

I bumped my PR that way and now I see more packit builds \o/ .

pmoravec added a commit to pmoravec/sos that referenced this pull request Dec 17, 2023
Relevant: sosreport#3447
Relevant: sosreport#3448

Signed-off-by: Pavel Moravec <pmoravec@redhat.com>
@dnegreira
Copy link
Contributor

Hi @arif-ali and @TrevorBenson,

@dnegreira FYI, does that sound reasonable for the control file, or do we want to avoid doing a Depends on a universe package, see below?
@arif-ali Would adding a Recommends and placing it there be more appropriate in case Universe is not enabled?

Unfortunately, both recommendations are not allowed as sosreport is in Ubuntu's main repositories.

The condition to load boto3, if it is available, LGTM as a way forward.

TrevorBenson pushed a commit to TrevorBenson/sos that referenced this pull request Dec 18, 2023
Relevant: sosreport#3447
Relevant: sosreport#3448

Signed-off-by: Pavel Moravec <pmoravec@redhat.com>
@TrevorBenson
Copy link
Member Author

I got a BAD key ID on F38, I originally suspected it was related to this. However I didn't get any key issues on F39 when installing.

I didn't check if F39 had an additional change and it now uses something only compatible with >=F39. Just letting you know in case you want to drop F38 since it does not appear to be testable.

@TrevorBenson
Copy link
Member Author

TrevorBenson commented Dec 23, 2023

I think this may be ready to convert from draft to review. I've tested sos report with various option combinations with (and without):

  • --upload-protocol
  • --upload-url
  • --upload-s3-region
  • --upload-s3-bucket
  • --upload-s3-endpoint
  • --upload-s3-access-key
  • --upload-s3-secret-key

For my F39 workstation/laptop I am having some issues with testing sos collect but I have an el8-based lab I should be able to try it on and believe I have previously used the collector --transport saltstack with. So believe I should be able to run some additional tests to confirm it works for collections as well.

I do however welcome anyone with interest in uploads to s3 buckets to give a Packit RPM a try in case I overlooked anything by performing each test manually.

TrevorBenson pushed a commit to TrevorBenson/sos that referenced this pull request Dec 23, 2023
As Fedora 39 is out, let bump its versions in copr builds and GCE
testing.

Sadly, GCE images offer F39 beta only, so far..

Resolves: sosreport#3448
Relevant: sosreport#3447

Signed-off-by: Pavel Moravec <pmoravec@redhat.com>
TrevorBenson pushed a commit to TrevorBenson/sos that referenced this pull request Dec 23, 2023
Relevant: sosreport#3447
Relevant: sosreport#3448

Signed-off-by: Pavel Moravec <pmoravec@redhat.com>
@TrevorBenson
Copy link
Member Author

Enterprise Linux

Repositories

@pmoravec Commit 437fe69 extends the .packit.yaml to include centos as I suspect CentOS Stream will be the intended repository used for testing on Red Hat 8/9 and downstream recompiles. As the subject of #3448 is specific to Fedora should I open a new PR specific to adding CentOS, or would you prefer to include these changes in the existing PR?

Testing

The 4.6.0-5 current el8 RPM was newer than the 4.6.0-1 from COPR. I suspect this may be an artifact due to Red Hat backporting fixes. I couldn't seem to target it specifically as a downgrade either:

dnf downgrade 4.6.0-1.20231223172713522168.pr3447.87.g0887f9f5.el8
Last metadata expiration check: 0:13:50 ago on Sat 23 Dec 2023 01:46:10 AM PST.
No package 4.6.0-1.20231223172713522168.pr3447.87.g0887f9f5.el8 available.
Error: No packages marked for downgrade.

In any case, I was able to avoid removing the dependencies and get it installed via:

rpm -e --nodeps sos
dnf install --disablerepo=baseos sos

However, given the simplicity of the Packit suggestion I'm not sure this is the intended way to install on EL distros w/ COPR, or just the quick workaround I came up with. Did I overlook an option to Packit that would have based the RPM off the current upstream NEVRA when generating the RPM for that COPR repository?


I'll clean up this PR by dropping cherry-picks, or commits moving to a new PR, before converting this from Draft to Ready once I know your preference.

Thanks

@TrevorBenson TrevorBenson force-pushed the upload-to-s3-buckets branch 2 times, most recently from 2c55a4d to 0a0a158 Compare December 25, 2023 20:37
@TrevorBenson TrevorBenson marked this pull request as ready for review December 25, 2023 20:37
@TrevorBenson
Copy link
Member Author

@arif-ali Do actions/workflows build a Debian PKG for testing? I've tested Fedora and Enterprise Linux with success. I have not tested on Ubuntu distro's though since there were no suggestions similar to Packit.

@arif-ali
Copy link
Member

@arif-ali Do actions/workflows build a Debian PKG for testing? I've tested Fedora and Enterprise Linux with success. I have not tested on Ubuntu distro's though since there were no suggestions similar to Packit.

Yes it does, below is the deb artifact from the cirrus builds and was built on Ubuntu 23.10 (mantic), which you could try

https://api.cirrus-ci.com/v1/artifact/task/5584901588647936/packages/sos_cirrus.deb

Alternatively, a snap is also created, and below is the link to the snap that as created as part of the build

https://api.cirrus-ci.com/v1/artifact/task/6710801495490560/packages/sosreport_test_amd64.snap

Copy link
Member

@TurboTurtle TurboTurtle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall it looks good from a code perspective, some minor notes below.

I'm kind of torn on the Recommends - I fully get the intent (and certainly not having it as a hard Requires), but afaik most rpm-based distros default to "always install recommends packages" and I wonder if pulling in boto3 as, effectively, a default is the right choice.

sos/collector/__init__.py Show resolved Hide resolved
sos/policies/distros/__init__.py Outdated Show resolved Hide resolved
sos/policies/distros/__init__.py Outdated Show resolved Hide resolved
sos/policies/distros/__init__.py Outdated Show resolved Hide resolved
sos/policies/distros/__init__.py Show resolved Hide resolved
sos/policies/distros/__init__.py Outdated Show resolved Hide resolved
@TrevorBenson
Copy link
Member Author

TrevorBenson commented Jan 8, 2024

Overall it looks good from a code perspective, some minor notes below.

I'm kind of torn on the Recommends - I fully get the intent (and certainly not having it as a hard Requires), but afaik most rpm-based distros default to "always install recommends packages" and I wonder if pulling in boto3 as, effectively, a default is the right choice.

I'll make whatever change is suggested. However, isn't the benefit of using Recommends instead of Requires that the end user could bypass this package altogether when using --setopt=install_weak_deps=False and avoid potential installation failures in the event the RPM is not available? While the latter should be rare, unless repositories are being recreated without complete package sets, I have seen this and deal with it almost daily.

@TurboTurtle
Copy link
Member

Overall it looks good from a code perspective, some minor notes below.
I'm kind of torn on the Recommends - I fully get the intent (and certainly not having it as a hard Requires), but afaik most rpm-based distros default to "always install recommends packages" and I wonder if pulling in boto3 as, effectively, a default is the right choice.

I'll make whatever change is suggested. However, isn't the benefit of using Recommends instead of Requires that the end user could bypass this package altogether when using --setopt=install_weak_deps=False and avoid potential installation failures in the event the RPM is not available? While the latter should be rare, unless repositories are being recreated without complete package sets, I have seen this and deal with it almost daily.

Oh I'm almost certainly being too paranoid about this :).

My new job has just happened to push package set security (i.e. "as little as possible") to the front of my mind at this point in time, that's all.

Taking a bit of a step back from that, you're right. Anyone who actually cares about this will either know about weak deps or have already disabled them. No issue from me on the spec file anymore.

@TrevorBenson
Copy link
Member Author

@arif-ali @TurboTurtle Just as an FYI I'd prefer this hold off from merging until after #3463 is finalized/merged and I can include an additional commit extending this branch to include --upload-s3-secret-key to the obfuscation. This new secret is what drove me investigate and to open #3462 in the first place.

Thanks!

@TrevorBenson
Copy link
Member Author

@arif-ali Do actions/workflows build a Debian PKG for testing? I've tested Fedora and Enterprise Linux with success. I have not tested on Ubuntu distro's though since there were no suggestions similar to Packit.

Yes it does, below is the deb artifact from the cirrus builds and was built on Ubuntu 23.10 (mantic), which you could try

https://api.cirrus-ci.com/v1/artifact/task/5584901588647936/packages/sos_cirrus.deb

Alternatively, a snap is also created, and below is the link to the snap that as created as part of the build

https://api.cirrus-ci.com/v1/artifact/task/6710801495490560/packages/sosreport_test_amd64.snap

Seems I forgot to report back on this the other week.

@arif-ali I tested both the deb package and the snap (using --classic --dangerous). As long as the system had the python3-boto3 package installed the s3 upload worked as intended.

@TrevorBenson
Copy link
Member Author

TrevorBenson commented Jan 15, 2024

The force push was for the rebase on main to include #3463 so that a5b9539 could be added.

@TrevorBenson
Copy link
Member Author

@TurboTurtle @arif-ali
Does the rpm Build From Checkout - centos-stream-8 failure (due to timeout) need to be addressed? I don't see a workflow dispatch or an option to rerun failed tests.

If not I believe this PR is ready for final reviews.

@TurboTurtle
Copy link
Member

I restarted the job - since this particular build is done via Cirrus only Pavel, Arif, Bryn, and myself have permissions to restart it (since the permission handling on non GH-enterprise orgs like us is very coarse - you have to be an "owner" to do it here).

The timeout is definitely not related to the changes here, but it does block the other cirrus jobs from running, so I'll watch this and restart it again if needed.

Copy link
Member

@TurboTurtle TurboTurtle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code looks good to me, just a stickler point from our Contribution Guidelines. The commit messages should be prefixed with a relevant tag, e.g. [upload] or somesuch (it makes trawling the git history much easier as time goes on). Also the second commit should be squashed into the first. The S3 obfuscation commit is perfectly fine being a separate commit in the chain.

@TrevorBenson
Copy link
Member Author

Code looks good to me, just a stickler point from our Contribution Guidelines. The commit messages should be prefixed with a relevant tag, e.g. [upload] or somesuch (it makes trawling the git history much easier as time goes on). Also the second commit should be squashed into the first. The S3 obfuscation commit is perfectly fine being a separate commit in the chain.

Added the context [upload] and squashed.

@pmoravec
Copy link
Contributor

I just dont get the ubuntu-18.4 CI test error:

Could not initialize 'report': future feature annotations is not defined (__init__.py, line 5)

We dont use annotations from future class, esp. we dont import that on line 5. Logs from the test are very concise..

@arif-ali
Copy link
Member

I just dont get the ubuntu-18.4 CI test error:

Could not initialize 'report': future feature annotations is not defined (__init__.py, line 5)

We dont use annotations from future class, esp. we dont import that on line 5. Logs from the test are very concise..

I'll need to check this locally, and see what is going on.

Ultimately, the test is being done, once the snap is installed and the snap is based on 22.04's python, and doesn't use the python on 18.04. I will get back on this one, and report back on why this is not working

@arif-ali
Copy link
Member

On that, below is the error we get on this particular patch, it's not obvious where the issue is, and remember debugging something very similar early last year, I will continue to debug, and get back on the resolution

root@bionic-sos-snap-test:~/sos# sos collect --list-options --debug
Could not initialize 'collect': future feature annotations is not defined (__init__.py, line 5)
Traceback (most recent call last):
  File "/snap/sosreport/x1/bin/sos", line 21, in <module>
    sos = SoS(sys.argv[1:])
  File "/root/sos/sos/__init__.py", line 124, in __init__
    self._init_component()
  File "/root/sos/sos/__init__.py", line 189, in _init_component
    raise err
  File "/root/sos/sos/__init__.py", line 184, in _init_component
    self._component = _to_load(self.parser, self.args, self.cmdline)
  File "/root/sos/sos/collector/__init__.py", line 141, in __init__
    super(SoSCollector, self).__init__(parser, parsed_args, cmdline_args)
  File "/root/sos/sos/component.py", line 90, in __init__
    self.load_local_policy()
  File "/root/sos/sos/component.py", line 132, in load_local_policy
    probe_runtime=self.load_probe)
  File "/root/sos/sos/policies/__init__.py", line 36, in load
    import sos.policies.distros
  File "/root/sos/sos/policies/distros/__init__.py", line 30, in <module>
    import requests
  File "/usr/lib/python3/dist-packages/requests/__init__.py", line 43, in <module>
    import urllib3
  File "/snap/sosreport/x1/lib/python3.10/site-packages/urllib3/__init__.py", line 5
    from __future__ import annotations
    ^
SyntaxError: future feature annotations is not defined

snap/snapcraft.yaml Outdated Show resolved Hide resolved
Signed-off-by: Trevor Benson <trevor.benson@gmail.com>
Signed-off-by: Trevor Benson <trevor.benson@gmail.com>
@TurboTurtle TurboTurtle merged commit ade3a8a into sosreport:main Jan 19, 2024
35 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

--upload-protocol for s3
5 participants