[several] Ubuntu 24.04 support + various improvements #246

SwooshyCueb · 2024-05-01T23:38:52Z

Addresses #89
Addresses #236
Addresses #237
Addresses #238
Addresses #239
Addresses #240
Addresses #241
Addresses #242
Addresses #243
Addresses #244
Addresses #245
Addresses #248
In service of irods/irods#7592

Companion PRs:

[several] Ubuntu 24.04 support + various improvements (main) irods#7733
[several] Ubuntu 24.04 support + various improvements (4-3-stable) irods#7739
[irods/irods#7592] Fix centos 7 build with new externals (main) irods_client_icommands#477
[irods/irods#7592] Fix centos 7 build against libstdc++ with new externals (4-3-stable) irods_client_icommands#478
[irods/irods#7592] add ubuntu 24.04 projects irods_testing_environment#210
[multiple] Update for externals changes + Ubuntu 24.04 externals builder + various improvements irods_development_environment#137

Notes:

Ubuntu 24.04 uses GCC 13 and libstdc++ 13 by default.
In order to build clang 13 against GCC 13 and libstdc++ 13, some patching is needed that is too complex to handle with basic shell commands, so this PR introduces new patching functionality.
Additionally, a couple of the nlohmann-json tests fail to build. This is fixed upstream by patching the tests themselves. This is fixed in this PR by simply turning off the tests.
Testing will be more thorough that would normally be warranted for adding a distro, as this PR patches many externals packages including the compiler and libc++.
This PR introduces a python module distro_info.py which is likely to make its way over to the main irods repository soon-ish.

install_prerequisites.py

distro_info.py

patches/nanodbc/1001-add-missing-limits-include.patch

build.py

distro_info.py

SwooshyCueb · 2024-05-23T19:51:47Z

Tests are passing! This PR is ready for review.

korydraughn · 2024-05-23T21:39:52Z

Good stuff. May not be able to look at this until after UGM.

alanking

Looks great. I just noticed a couple of things in the new module.

Also, would it be good to include some light instructions regarding the patch system? Maybe this is a common thing with which I am unfamiliar, but it seems like it would be good to leave the process for finding / applying patches for posterity. We can also create an issue for adding that to the README (similar to the directions about bumping build number versus bumping build version). Just a thought.

distro_info.py

install_prerequisites.py

korydraughn · 2024-06-10T22:02:47Z

Good stuff.

alanking

Seems good. Please await another approval and I think we're good to # it.

A couple of follow-up questions:

Do we need to release new externals packages before merging the companion PRs?
Had a question about this from OP: "This PR introduces a python module distro_info.py which is likely to make its way over to the main irods repository soon-ish." How do we feel about maintaining a PyPi package? Is that too much? Just thinking ahead to the duplicate code question... (one here, one in main repo)

Neither of these questions are blockers for merging, but just figured I'd put them here.

SwooshyCueb · 2024-06-11T21:03:12Z

Do we need to release new externals packages before merging the companion PRs?

Only the ones that are still drafts.

Had a question about this from OP: "This PR introduces a python module distro_info.py which is likely to make its way over to the main irods repository soon-ish." How do we feel about maintaining a PyPi package? Is that too much? Just thinking ahead to the duplicate code question... (one here, one in main repo)

I don't think that's a solution. If everything goes to plan, it's going to be a development (and possibly runtime) dependency of irods. Therefore, if it's not part of the repository, it has to be installable from a package we can declare a dependency on. We can't declare dependencies on PyPi packages, which is why we have the special handling for pyodbc on centos 7.

korydraughn · 2024-06-11T21:55:06Z

Is there a possibility for a patch to result in a successful build of externals, but break something within the iRODS server?

Yes. That's why I did so many runs of the test suite.

Did your testing involve all platforms supported by iRODS? My guess is yes due to the nature of the PR. What about plugins?

If there turns out to be an issue caused by a patch, how do you imagine we address them? Have you considered that?

SwooshyCueb · 2024-06-11T23:58:28Z

Did your testing involve all platforms supported by iRODS? My guess is yes due to the nature of the PR. What about plugins?

I ran the core tests on Ubuntu 22.04 and Ubuntu 24.04 with iRODS built from [several] Ubuntu 24.04 support + various improvements (main) irods#7733 and externals built from this PR
I ran the core tests on Ubuntu 22.04 and Ubuntu 24.04 with iRODS built from [several] Ubuntu 24.04 support + various improvements (4-3-stable) irods#7739 and externals built from this PR
I ran the core tests on Ubuntu 22.04 with iRODS 4.3.2 (from the apt repo, not my own build, so tests relying on optional microservices were not run) and externals built from this PR

I did not run the test suite on any other platforms, nor did I run any plugin test suites. I did, however, ensure that everything still built on all platforms.

If there turns out to be an issue caused by a patch, how do you imagine we address them? Have you considered that?

Depending on the situation, more than likely one of the following:

Remove the patch
Replace the patch
Fix the patch
Add another patch (this one might seem weird, but for externally-sourced patches, it can be the right move)

A new externals package would need to be published with the updated patch set.

alanking · 2024-06-12T14:57:29Z

A new externals package would need to be published with the updated patch set.

And for cases where the patch is fixing a broken external, I imagine this would be a package_revision bump rather than a consortium_build_number bump so that the broken package(s) would be replaced. Does that seem right?

SwooshyCueb · 2024-06-12T15:02:10Z

A new externals package would need to be published with the updated patch set.

And for cases where the patch is fixing a broken external, I imagine this would be a package_revision bump rather than a consortium_build_number bump so that the broken package(s) would be replaced. Does that seem right?

Generally speaking, yes

korydraughn

If you're happy with changes, squash to taste. No pounds.

SwooshyCueb · 2024-06-12T15:45:38Z

I wasn't planning on squashing any more than I already have

korydraughn

Pound it.

…EFIX`

SwooshyCueb · 2024-06-12T16:19:42Z

#'d

trel reviewed May 1, 2024

View reviewed changes

install_prerequisites.py Outdated Show resolved Hide resolved

distro_info.py Outdated Show resolved Hide resolved

patches/nanodbc/1001-add-missing-limits-include.patch Outdated Show resolved Hide resolved

SwooshyCueb force-pushed the noble.main branch 7 times, most recently from 76b5590 to 0ea64b6 Compare May 3, 2024 06:30

SwooshyCueb mentioned this pull request May 3, 2024

[multiple] Update for externals changes + Ubuntu 24.04 externals builder + various improvements irods/irods_development_environment#137

Merged

SwooshyCueb changed the title ~~[WIP] Ubuntu 24.04 support + various improvements~~ [several] Ubuntu 24.04 support + various improvements May 3, 2024

SwooshyCueb commented May 3, 2024

View reviewed changes

build.py Outdated Show resolved Hide resolved

SwooshyCueb force-pushed the noble.main branch 2 times, most recently from f66560a to e83e26d Compare May 3, 2024 20:01

SwooshyCueb mentioned this pull request May 4, 2024

[several] Ubuntu 24.04 support + various improvements (main) irods/irods#7733

Merged

SwooshyCueb force-pushed the noble.main branch 3 times, most recently from f1e627c to 7224e24 Compare May 9, 2024 04:23

SwooshyCueb commented May 9, 2024

View reviewed changes

distro_info.py Outdated Show resolved Hide resolved

SwooshyCueb force-pushed the noble.main branch 2 times, most recently from 2619ebc to 5ba1255 Compare May 9, 2024 04:39

SwooshyCueb mentioned this pull request May 22, 2024

[irods/irods#7592] add ubuntu 24.04 projects irods/irods_testing_environment#210

Merged

SwooshyCueb marked this pull request as ready for review May 23, 2024 19:51

SwooshyCueb force-pushed the noble.main branch from 5ba1255 to 5f3e210 Compare May 24, 2024 16:22

alanking reviewed Jun 10, 2024

View reviewed changes

distro_info.py Outdated Show resolved Hide resolved

distro_info.py Outdated Show resolved Hide resolved

distro_info.py Outdated Show resolved Hide resolved

install_prerequisites.py Show resolved Hide resolved

SwooshyCueb mentioned this pull request Jun 10, 2024

Document new patch system #251

Open

SwooshyCueb force-pushed the noble.main branch from 5f3e210 to 0a98f0c Compare June 11, 2024 19:15

alanking approved these changes Jun 11, 2024

View reviewed changes

korydraughn reviewed Jun 12, 2024

View reviewed changes

korydraughn approved these changes Jun 12, 2024

View reviewed changes

alanking approved these changes Jun 12, 2024

View reviewed changes

SwooshyCueb added 15 commits June 12, 2024 11:18

[irods#238] Remove mkdir_p methods

7c93cb5

[irods#243] install_prerequisites.py: Do not dnf/yum update

7cd2a0b

[irods#240] Remove old SUSE and OSX support code

58d6f27

[irods#241] Consolidate package lists

b082997

[irods#245] Add python artifacts to .gitignore

34f6e88

[irods#242] Install curl-minimal on EL9+

9269d92

[irods#239] Install cmake

d19455c

[irods#236] Remove unneeded steps from README for EL

4112949

[irods#244] Introduce new module distro_info.py

4f04771

[irods#237] README.md: instruct user to set `IRODS_EXTERNALS_GCC_PR…

db92503

…EFIX`

[irods#89] Do not install patchelf

8631d82

[irods/irods#7592] Add new source patch functionality

b0c33d7

[irods#248] Add distro patches

5a9910f

[irods/irods#7592] Do not build nlohmann-json tests

5144be6

[irods/irods#7592] Add Ubuntu 24.04 support

980e7f2

SwooshyCueb force-pushed the noble.main branch from 0a98f0c to 980e7f2 Compare June 12, 2024 16:19

alanking merged commit 0040b5e into irods:main Jun 12, 2024
1 check failed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[several] Ubuntu 24.04 support + various improvements #246

[several] Ubuntu 24.04 support + various improvements #246

SwooshyCueb commented May 1, 2024 •

edited

SwooshyCueb commented May 23, 2024

korydraughn commented May 23, 2024

alanking left a comment

korydraughn commented Jun 10, 2024

alanking left a comment

SwooshyCueb commented Jun 11, 2024 •

edited

korydraughn commented Jun 11, 2024

SwooshyCueb commented Jun 11, 2024

alanking commented Jun 12, 2024

SwooshyCueb commented Jun 12, 2024

korydraughn left a comment

SwooshyCueb commented Jun 12, 2024

korydraughn left a comment

SwooshyCueb commented Jun 12, 2024

[several] Ubuntu 24.04 support + various improvements #246

[several] Ubuntu 24.04 support + various improvements #246

Conversation

SwooshyCueb commented May 1, 2024 • edited

SwooshyCueb commented May 23, 2024

korydraughn commented May 23, 2024

alanking left a comment

Choose a reason for hiding this comment

korydraughn commented Jun 10, 2024

alanking left a comment

Choose a reason for hiding this comment

SwooshyCueb commented Jun 11, 2024 • edited

korydraughn commented Jun 11, 2024

SwooshyCueb commented Jun 11, 2024

alanking commented Jun 12, 2024

SwooshyCueb commented Jun 12, 2024

korydraughn left a comment

Choose a reason for hiding this comment

SwooshyCueb commented Jun 12, 2024

korydraughn left a comment

Choose a reason for hiding this comment

SwooshyCueb commented Jun 12, 2024

SwooshyCueb commented May 1, 2024 •

edited

SwooshyCueb commented Jun 11, 2024 •

edited