mongodb: fixes build and sanitize package #172009

bryanasdev000 · 2022-05-08T01:11:57Z

Description of changes

This should close #171928 and #155121.

Basically a umbrella PR to fix MongoDB issues on Nixpkgs.

With help from @andersk @bachp @kfiz @superherointj @Artturin, thanks a lot!

Things done

pkgs/servers/nosql/mongodb/v4_0.nix

pkgs/top-level/all-packages.nix

kfiz · 2022-05-09T06:08:54Z

Please change this
stdenv = gcc10Stdenv;
in all-packages.nix
to
stdenv = if stdenv.cc.isGNU then gcc10Stdenv else stdenv;
or otherwise we run into problems on non-linux

bryanasdev000 · 2022-05-09T14:02:31Z

stdenv = if stdenv.cc.isGNU then gcc10Stdenv else stdenv;

Done!

bryanasdev000 · 2022-05-09T15:18:43Z

Folks, I think it's ready to ship, a squash is necessary?

I am just waiting for ofborg output before removing the draft.

CC: @andersk @bachp @kfiz @superherointj

kfiz · 2022-05-09T15:25:43Z

Folks, I think it's ready to ship, a squash is necessary?

I am just waiting for ofborg output before removing the draft.

CC: @andersk @bachp @kfiz @superherointj

LGTM. To nitpick, there is a minor typo in the doc sections: "provider" -> "provide".

bryanasdev000 · 2022-05-09T15:37:19Z

provider

Fixed, will probably need to squash :P

thiagokokada · 2022-05-09T21:55:57Z

@ofborg eval

bryanasdev000 · 2022-05-09T22:03:32Z

Need to fix top level entry...

bryanasdev000 · 2022-07-04T04:41:19Z

So, I pushed a mess of tests, so far, MongoDB 4.4 and 5.0.9 still fails to build.

Will try it again when less sleepy, gist for logs https://gist.github.com/bryanasdev000/b086bbf9c9a0f278b9f6eb4b1670549e

bryanasdev000 · 2022-07-04T05:20:06Z

Will try to build PR #178820 over master to check if it builds and what's the difference that's causing it to fail.

EDIT: Fails too https://gist.github.com/bryanasdev000/b086bbf9c9a0f278b9f6eb4b1670549e#file-5-0-7-txt

Et7f3 · 2022-07-07T18:10:12Z

I tested on a Mac builder (with better CPU and more disk). It was also broken. It seem the issue was boost 1.79 you might be interested in cherry-pick #180620

bryanasdev000 · 2022-07-13T08:40:52Z

I tested on a Mac builder (with better CPU and more disk). It was also broken. It seem the issue was boost 1.79 you might be interested in cherry-pick #180620

Sorry for the delay @Et7f3, been busy with IRL.

I've tried to build 4.4, and I am leaving 5.0 in the queue to build on my laptop, but so far 4.4 fails with iostream errors on your branch and another full set of errors with cherry-picking (sadly did not catch the full log).

If mongo 4.4 and/or 5.0 is working for you on Darwin, I suggest that you unify your PRs (#178820, #178820 and any other necessary piece to make it fully work) and push it to review, so this PR will not be a blocker. Also, I can give some help in code review at least.

Note that I only have access to x86-64 architectures, so we may be chasing different errors :P

Et7f3 · 2022-07-13T08:48:40Z

I don't think I will merge my two PR since it is targeted for 2 users classes. (Maybe their is overlap). From your logs it seem mongodb include special file only for linux hence it didn't caused issue on the darwin build.

Et7f3 · 2022-07-20T00:37:32Z

Ok I did a round on linux and finally merged the 2 PR (since their is a overlap). See my PR for darwin and it should work for darwin and linux. (I am currently rebuilding after a rebase to be sure nothing got broken in the 2k commits)

SuperSandro2000 · 2022-09-10T19:39:44Z

pkgs/top-level/all-packages.nix

    sasl = cyrus_sasl;
    boost = boost17x.override { enableShared = false; };
    inherit (darwin) cctools;
    inherit (darwin.apple_sdk.frameworks) CoreFoundation Security;
  };

-  mongodb-5_0 = callPackage ../servers/nosql/mongodb/5.0.nix {
+  mongodb-5_0 = callPackage ../servers/nosql/mongodb/v5_0.nix {
    sasl = cyrus_sasl;


Do we even need this? It seems always to be cyrus_sasl.

Would it be clear which sasl library should be used if we removed it? There seem to be at least 2 options, namely cyrus_sasl and gsasl.

Does mongodb even work with gsasl? If so this should rather be a boolean input to enable one or the other.

I think it would be possible to use gsasl:
https://www.mongodb.com/docs/manual/reference/program/mongos/#std-option-mongos.--ldapBindSaslMechanisms

kfiz · 2022-09-11T08:29:16Z

Ok I did a round on linux and finally merged the 2 PR (since their is a overlap). See my PR for darwin and it should work for darwin and linux. (I am currently rebuilding after a rebase to be sure nothing got broken in the 2k commits)

@Et7f3 @bryanasdev000 Should we try to wrap this up?

Et7f3 · 2022-09-11T08:39:21Z

I tested my PR #180620 on both linux and mac 2 months ago and all builded. I didn't retested recently but it should still be in good shape. If you @kfiz mind reviewing. It is a full PR done.

bryanasdev000 · 2022-09-11T09:27:42Z

I am away from my computer now, but I can try to dust it off and try again once I get back home.

We have 3 PRs now trying to fix Mongo at nixpkgs, this, @Et7f3 one and #190548 maybe now we can fix it for real.

bryanasdev000 · 2022-09-11T09:29:27Z

I tested my PR #180620 on both linux and mac 2 months ago and all builded. I didn't retested recently but it should still be in good shape. If you @kfiz mind reviewing. It is a full PR done.

@Et7f3 since you have hardware for it, could you please retest atop of master? Just in case, to test Darwin specifics.

kfiz · 2022-09-11T09:52:10Z

I tested my PR #180620 on both linux and mac 2 months ago and all builded. I didn't retested recently but it should still be in good shape. If you @kfiz mind reviewing. It is a full PR done.

I tested the v5 patch on darwin and that worked. I'll have a look at the other adjustments later, but would give @bryanasdev000 the lead here as he has the best overview.

If we think we are good to go, we can consider updating the v5 build to 5.0.12 which is the latest patch release. I have a version of that here kfiz@b48aa95

SuperSandro2000 · 2022-09-11T15:22:43Z

nixos/modules/services/databases/mongodb.nix

+     mkDefault (if versionAtLeast config.system.stateVersion "22.11" then pkgs.mongodb-5_0
+           else if versionAtLeast config.system.stateVersion "22.05" then pkgs.mongodb-3_4
+           else throw "please set your desired package for state version ${config.system.stateVersion}");


Suggested change

mkDefault (if versionAtLeast config.system.stateVersion "22.11" then pkgs.mongodb-5_0

else if versionAtLeast config.system.stateVersion "22.05" then pkgs.mongodb-3_4

else throw "please set your desired package for state version ${config.system.stateVersion}");

mkDefault (if versionAtLeast config.system.stateVersion "22.11" then pkgs.mongodb-5_0

else pkgs.mongodb-3_4;

SuperSandro2000 · 2022-09-11T15:23:16Z

nixos/modules/services/databases/mongodb.nix

@@ -32,8 +32,7 @@ in
      enable = mkEnableOption "the MongoDB server";

      package = mkOption {
-        default = pkgs.mongodb;


We should keep a default package

But this option will be overridden with the condition of stateVersion (a little bit below this snippet), so why define a version here? Also, if defined, won't conflict with this same condition?

SuperSandro2000 · 2022-09-11T15:24:15Z

nixos/doc/manual/release-notes/rl-2211.section.md

@@ -92,6 +92,8 @@ In addition to numerous new and upgraded packages, this release has the followin
  as it requires `qt4`, which reached its end-of-life 2015 and will no longer be supported by nixpkgs.
  [According to Barco](https://www.barco.com/de/support/knowledge-base/4380-can-i-use-linux-os-with-clickshare-base-units) many of their base unit models can be used with Google Chrome and the Google Cast extension.

+- MongoDB's packages were updated to each latest major version and versions older than 4.0 marked with `meta.known Vulnerabilities` to flag as EOLed by upstream yet still in tree to provide a clean upgrade path, also MongoDB 5.x is the default version from now on.


Suggested change

- MongoDB's packages were updated to each latest major version and versions older than 4.0 marked with `meta.known Vulnerabilities` to flag as EOLed by upstream yet still in tree to provide a clean upgrade path, also MongoDB 5.x is the default version from now on.

- MongoDB's packages were updated to each latest major version and versions older than 4.0 marked with `meta.knownVulnerabilities` to flag as EOLed by upstream yet still in tree to provide a clean upgrade path, also MongoDB 5.x is the default version from now on.

that is not quite correct. meta.knownVulnerabilities has nothing to do with the maintenance status of upstream bit that the package contains long standing not fixed vulnerabilities which is the case because those packages are EOL.

Also what is the upgrade path for mongodb? Do we really need to keep 4.0?

This is what we found to mark EOL and give space to users to do a clean upgrade from 3.4 to the latest (be it 5.X or 6.X if #190548 gets merged first).

There is another Nix mechanism to do that?

And yes we need to keep all versions, as one should upgrade version by version until it reaches the desired one, see https://www.mongodb.com/docs/manual/release-notes/5.0-upgrade-standalone/ and #155121 (comment) for reference.

Besides that, I think another way is doing the migration with a container then once the data is OK, start the service in NixOS with the desired version.

SuperSandro2000 · 2022-09-11T15:27:52Z

pkgs/servers/nosql/mongodb/mongodb.nix

@@ -133,7 +133,7 @@ in stdenv.mkDerivation rec {
    description = "A scalable, high-performance, open source NoSQL database";
    homepage = "http://www.mongodb.org";
    inherit license;
-
+    knownVulnerabilities =  if (versionAtLeast version "4.0") then [] else [ "EOLed version, please check https://www.mongodb.com/support-policy/lifecycles and official docs on how to upgrade" ];


Suggested change

knownVulnerabilities = if (versionAtLeast version "4.0") then [] else [ "EOLed version, please check https://www.mongodb.com/support-policy/lifecycles and official docs on how to upgrade" ];

knownVulnerabilities = lib.optionals (versionAtLeast version "4.0") [ "EOLed version, please check https://www.mongodb.com/support-policy/lifecycles and official docs on how to upgrade" ];

Please add concrete CVEs from https://www.mongodb.com/alerts#security into here. This field is not for marking EOL software without vulnerabilities.

Answered above.

bryanasdev000 · 2022-09-12T00:41:45Z

Ok, thanks to @Et7f3 and @kfiz we now should have all mongodb versions in tree building (with exception being the v6.x per #190548 (comment)) in Darwin, thanks a lot!

Now, I think we should merge PR #190548 first and then focus on this one with all needed decisions, that include:

Fix all builds on Linux
Chose what, when and how to deprecate EOL Mongo versions
Changelog
Fix conflicts
Release update (if any)
Refactors (if any)

This PR is also a little bit old and big, if needed we can split it in multiples, each one addressing some of the above points.

kfiz · 2022-09-12T11:21:18Z

This PR is also a little bit old and big, if needed we can split it in multiples, each one addressing some of the above points.
@SuperSandro2000 What would you suggest?

Et7f3 · 2022-10-12T16:43:10Z

mongodb v6.0 landed nixpkgs so we can revive this PR

bryanasdev000 · 2022-10-25T08:17:36Z

mongodb v6.0 landed nixpkgs so we can revive this PR

And thanks to @kfiz we should have MongoDB 6.x running on Darwin land soon.

I will try to check what is missing for close this PR in the weekend and maybe redo this PR or split it in multiple.

dotlambda · 2023-01-10T10:24:45Z

Can this be closed?

bryanasdev000 · 2023-02-03T12:12:46Z

Can this be closed?

I think we can, AFAIK we have both old and new versions working and mongodb 6 as default pkg.

What will be missing is to deprecate old versions (#195448) but honestly I am without energy to pursue this.

Thanks for the help everyone, if anyone need a bit of this PR, feel free to grab.

ofborg bot added 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 labels May 8, 2022

Artturin reviewed May 8, 2022

View reviewed changes

pkgs/servers/nosql/mongodb/v4_0.nix Outdated Show resolved Hide resolved

bryanasdev000 mentioned this pull request May 8, 2022

mongodb attribute should refer to a newer version #155121

Closed

bryanasdev000 changed the title ~~mongodb: fixes build~~ mongodb: fixes build and sanitize packages May 8, 2022

bryanasdev000 changed the title ~~mongodb: fixes build and sanitize packages~~ mongodb: fixes build and sanitize package May 8, 2022

ofborg bot requested review from offlinehacker, bluescreen303 and cstrahan May 8, 2022 04:55

bryanasdev000 force-pushed the mongodb-fixes branch from 972225d to e070a9b Compare May 8, 2022 06:49

bachp reviewed May 8, 2022

View reviewed changes

pkgs/top-level/all-packages.nix Outdated Show resolved Hide resolved

github-actions bot added 6.topic: nixos 8.has: changelog 8.has: documentation labels May 9, 2022

bryanasdev000 requested review from bachp, superherointj, andersk and Artturin May 9, 2022 15:19

bryanasdev000 force-pushed the mongodb-fixes branch from 2d9a65f to 4a23bfe Compare May 9, 2022 15:51

bryanasdev000 marked this pull request as ready for review May 9, 2022 15:51

bachp approved these changes May 9, 2022

View reviewed changes

superherointj approved these changes May 9, 2022

View reviewed changes

bryanasdev000 force-pushed the mongodb-fixes branch from 0c4110c to e56a391 Compare May 9, 2022 22:05

mongodb: backport openssl fix to 4.4

b2a54e1

mongodb: use default gcc version

f627a6f

ParetoOptimalDev mentioned this pull request Aug 19, 2022

mongosh package uses wrong node version v14.20.0 but needs >= 16.15.0 #187337

Closed

superherointj removed their request for review August 31, 2022 19:54

pacien mentioned this pull request Sep 10, 2022

mongodb-6_0: init at 6.0.1 #190548

Merged

13 tasks

SuperSandro2000 added the 2.status: merge conflict label Sep 10, 2022

SuperSandro2000 reviewed Sep 10, 2022

View reviewed changes

SuperSandro2000 reviewed Sep 11, 2022

View reviewed changes

pacien mentioned this pull request Oct 22, 2022

Remove end-of-life mongodb versions #195448

Open

bryanasdev000 closed this Feb 3, 2023

This pull request was closed.

	- MongoDB's packages were updated to each latest major version and versions older than 4.0 marked with `meta.known Vulnerabilities` to flag as EOLed by upstream yet still in tree to provide a clean upgrade path, also MongoDB 5.x is the default version from now on.
	- MongoDB's packages were updated to each latest major version and versions older than 4.0 marked with `meta.knownVulnerabilities` to flag as EOLed by upstream yet still in tree to provide a clean upgrade path, also MongoDB 5.x is the default version from now on.

	knownVulnerabilities = if (versionAtLeast version "4.0") then [] else [ "EOLed version, please check https://www.mongodb.com/support-policy/lifecycles and official docs on how to upgrade" ];
	knownVulnerabilities = lib.optionals (versionAtLeast version "4.0") [ "EOLed version, please check https://www.mongodb.com/support-policy/lifecycles and official docs on how to upgrade" ];

mongodb: fixes build and sanitize package #172009

mongodb: fixes build and sanitize package #172009

Conversation

bryanasdev000 commented May 8, 2022 • edited Loading

Description of changes

Things done

kfiz commented May 9, 2022

bryanasdev000 commented May 9, 2022

bryanasdev000 commented May 9, 2022

kfiz commented May 9, 2022

bryanasdev000 commented May 9, 2022

thiagokokada commented May 9, 2022

bryanasdev000 commented May 9, 2022

bryanasdev000 commented Jul 4, 2022

bryanasdev000 commented Jul 4, 2022 • edited Loading

Et7f3 commented Jul 7, 2022 • edited Loading

bryanasdev000 commented Jul 13, 2022

Et7f3 commented Jul 13, 2022

Et7f3 commented Jul 20, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kfiz commented Sep 11, 2022

Et7f3 commented Sep 11, 2022

bryanasdev000 commented Sep 11, 2022

bryanasdev000 commented Sep 11, 2022

kfiz commented Sep 11, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SuperSandro2000 Sep 11, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bryanasdev000 commented Sep 12, 2022 • edited Loading

kfiz commented Sep 12, 2022

Et7f3 commented Oct 12, 2022

bryanasdev000 commented Oct 25, 2022 • edited Loading

dotlambda commented Jan 10, 2023

bryanasdev000 commented Feb 3, 2023

bryanasdev000 commented May 8, 2022 •

edited

Loading

bryanasdev000 commented Jul 4, 2022 •

edited

Loading

Et7f3 commented Jul 7, 2022 •

edited

Loading

SuperSandro2000 Sep 11, 2022 •

edited

Loading

bryanasdev000 commented Sep 12, 2022 •

edited

Loading

bryanasdev000 commented Oct 25, 2022 •

edited

Loading