Infinite blockage when version cant be downloaded

[dcurletti]

No description provided.

[AJRdev]

@dcurletti Normally we already handle MongoDB v4.2 in this package, do you experience any issues with this particular version ?
If not this issue will be closed 🙂

[duro]

@AJRdev I am also experiencing issues getting the package to work with Mongo 4.2. The issue I am having is that when I specify that I want 4.2 like so in the setup:

mongoServer = new MongoMemoryReplSet({
  binary: {
    version: '4.2.0'
  },
  replSet: { storageEngine: 'wiredTiger' },
})

This works just fine locally on my Mac, however when this is running in CircleCI I get the following error:

   console.warn ../../node_modules/mongodb-memory-server-core/lib/util/MongoBinaryDownloadUrl.js:186
     Unknown linux distro Raspbian, falling back to legacy MongoDB build
 Downloading MongoDB 4.2.0: NaN % (0mb / NaNmb)
   console.error ../../node_modules/jest-jasmine2/build/jasmine/Env.js:289
     Unhandled error
   console.error ../../node_modules/jest-jasmine2/build/jasmine/Env.js:290
     Error: Too small (243 bytes) mongod binary downloaded from https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-4.2.0.tgz

The very odd thing is, I am running this in a monorepo which has many other services being tested that are using mongodb-memory-server, but are still just using the default version of 4.0.3 and even on CircleCI, those work just fine.

It is only the service trying to use 4.2 which fails on Circle.

The other using 4.0.3 still throw the warning: Unknown linux distro Raspbian, falling back to legacy MongoDB build, but the successfully download that fallback version just fine.

[kunalbhatt]

@AJRdev I am also experiencing issues getting the package to work with Mongo 4.2. The issue I am having is that when I specify that I want 4.2 like so in the setup:

mongoServer = new MongoMemoryReplSet({
  binary: {
    version: '4.2.0'
  },
  replSet: { storageEngine: 'wiredTiger' },
})

This works just fine locally on my Mac, however when this is running in CircleCI I get the following error:

   console.warn ../../node_modules/mongodb-memory-server-core/lib/util/MongoBinaryDownloadUrl.js:186
     Unknown linux distro Raspbian, falling back to legacy MongoDB build
 Downloading MongoDB 4.2.0: NaN % (0mb / NaNmb)
   console.error ../../node_modules/jest-jasmine2/build/jasmine/Env.js:289
     Unhandled error
   console.error ../../node_modules/jest-jasmine2/build/jasmine/Env.js:290
     Error: Too small (243 bytes) mongod binary downloaded from https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-4.2.0.tgz

The very odd thing is, I am running this in a monorepo which has many other services being tested that are using mongodb-memory-server, but are still just using the default version of 4.0.3 and even on CircleCI, those work just fine.

It is only the service trying to use 4.2 which fails on Circle.

The other using 4.0.3 still throw the warning: Unknown linux distro Raspbian, falling back to legacy MongoDB build, but the successfully download that fallback version just fine.

@duro you'll have to clear your node_modules cache locally for it to redownload the 4.2 to your local machine. The circle ci support for 4.2 is something that will have to be resolved

[duro]

@kunalbhatt Yea, I am not having trouble locally, just in CI. I actually worked around this by installing 4.2 into my CI image, and then using the systemBinary config value when running in CI. Not the ideal solution, but it at least got me moving.

[theonetheycallneo]

The $merge is a new command in 4.2

https://docs.mongodb.com/manual/reference/operator/aggregation/merge/

[hasezoey]

@duro is this still a problem in version 6.1?

[helt]

I can provide more information to this. I was facing this issue, because mongodb cleaned up their repos, and my version 4.0.3 was not available anymore. This caused a lot of pain, as it led to timeouts of the gitlab ci pipeline without any hint that a request to mongodb.org was basically blocking the pipeline. In the end, the gitlab ci pipeline just got canceld because reaching the upper bound of 1 hour.

This is for the git bash on windows.

In package.json I have:

    "devDependencies": {
        "mongodb-memory-server": "6.3.1",
    }
    "config": {
        "mongodbMemoryServer": {
            "debug": "1",
            "version": "4.3.3"
        }
    }

which did not work, but a

            "version": "4.2.3"

works.

I assume that the downloader does not treat a 403 response correctly (i.e. does not exit with error)

$ yarn run test
yarn run v1.22.0
$ jest --runInBand --setupFiles dotenv/config
  MongoMS:MongoMemoryServer Called MongoMemoryServer.ensureInstance() method: +0ms
  MongoMS:MongoMemoryServer  - no running instance, call `start()` command +1ms
  MongoMS:MongoMemoryServer Called MongoMemoryServer.start() method +1ms
  MongoMS:MongoMemoryServer Starting MongoDB instance with following options: {"port":63241,"dbName":"d10301ee-aced-464b-b094-dc2a9d711cd1","ip":"127.0.0.1","storageEngine":"ephemeralForTest","dbPath":"C:\\Users\\somebody\\AppData\\Local\\Temp\\mongo-mem-24388oM041E2TthBA","tmpDir":{"name":"C:\\Users\\somebody\\AppData\\Local\\Temp\\mongo-mem-24388oM041E2TthBA"},
"uri":"mongodb://127.0.0.1:63241/d10301ee-aced-464b-b094-dc2a9d711cd1?"} +69ms
  MongoMS:MongoBinary MongoBinary options: {"downloadDir":"\\node_modules\\.cache\\mongodb-memory-server\\mongodb-binaries","platform":"win32","arch":"x64","version":"4.3.3"} +0ms
  MongoMS:MongoBinaryDownloadUrl Using "mongodb-win32-x86_64-2008plus-ssl-4.3.3.zip" as the Archive String +0ms
  MongoMS:MongoBinaryDownloadUrl Using "https://fastdl.mongodb.org" as the mirror +2ms
  MongoMS:MongoBinaryDownload Downloading: https://fastdl.mongodb.org/win32/mongodb-win32-x86_64-2008plus-ssl-4.3.3.zip +0ms
  MongoMS:MongoBinaryDownload renamed C:\somwhere\node_modules\.cache\mongodb-memory-server\mongodb-binaries\mongodb-win32-x86_64-2008plus-ssl-4.3.3.zip.downloading to C:\somwhere\node_modules\.cache\mongodb-memory-server\mongodb-binaries\mongodb-win32-x86_64-2
008plus-ssl-4.3.3.zip +390ms
  MongoMS:MongoBinaryDownload extract(): C:\somwhere\node_modules\.cache\mongodb-memory-server\mongodb-binaries\4.3.3 +1ms
Downloading MongoDB 4.3.3: NaN % (0mb / NaNmb)
 RUNS  src/mongodb/init.spec.ts

somebody@pc2752 MINGW64 /somewhere (bug/fix-mongodb-memory-server)
$ curl -v https://fastdl.mongodb.org/win32/mongodb-win32-x86_64-2008plus-ssl-4.3.3.zip
> GET /win32/mongodb-win32-x86_64-2008plus-ssl-4.3.3.zip HTTP/2
> Host: fastdl.mongodb.org
> user-agent: curl/7.68.0
> accept: */*
>
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
} [5 bytes data]
< HTTP/2 403
< content-type: application/xml
< date: Tue, 03 Mar 2020 16:02:27 GMT
< server: AmazonS3
< x-cache: Error from cloudfront
< via: 1.1 9128c49d19c76fd86ec4c647434ccb0a.cloudfront.net (CloudFront)
< x-amz-cf-pop: FRA50-C1
< x-amz-cf-id: ZGB7XQXtUVXtkS6Wyd0I8LMIVAPfVjeQ5QsxGKt6YFmkXoJ7ukXNqA==
<
{ [243 bytes data]
100   243    0   243    0     0    589      0 --:--:-- --:--:-- --:--:--   591<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>4D5ABD4E16887D1E</RequestId><HostId>llWFJK1dun4p5d34mNyhLzEG9wjDcrW01+JOfezgwjnCUf03GfZz5osgW4S3zwaeepDC4/6Bee0=</HostId></Error>
* Connection #0 to host fastdl.mongodb.org left intact

[hasezoey]

@helt thanks for the extra insight, just to confirm, your github actions are windows too?

but yes, i also encountered "infinite blocking" too and this needs to be fixed

---

as for your platfrom (win32 - (sever) 2008 - plusssl) it seems like there are no versions above 4.0 anymore (reference)

---

I assume that the downloader does not treat a 403 response correctly

at the moment, it assumes its status code 200 - will be fixed in the next update

[hasezoey]

@helt i hope this got fixed by 6.3.2 (the infinite blockage)

[helt]

@helt thanks for the extra insight, just to confirm, your github actions are windows too?

github actions?

as for your platfrom (win32 - (sever) 2008 - plusssl) it seems like there are no versions above 4.0 anymore (reference)

well, 4.2.3 does it for me... on all platforms i tried: git bash windows, ubuntu WSL, docker debian10. All tests worked with that version.

I assume that the downloader does not treat a 403 response correctly

at the moment, it assumes its status code 200 - will be fixed in the next update

Awesome! That was quick!

[hasezoey]

github actions?

sorry i missred, i meant your gitlab pipeline - is that windows 2008 too?

well, 4.2.3 does it for me

for windows (server) i only see 4.2.3 for 2012 (not 2008)

[helt]

github actions?

sorry i missred, i meant your gitlab pipeline - is that windows 2008 too?

Ah, no no, that's Debian 10.

[hasezoey]

@helt this shouldnt happen (even before the fix), because for debian10 version 4.2.3 is available (reference search for debian10-4.2.3)

[helt]

oh boy. I chose 4.2.3, because it is available on all platforms i needed. As said in my initial comment, everything works with 4.2.3, but not with 4.3.3. See #226 (comment)

[hasezoey]

@helt sorry, again

but not with 4.3.3

searched again, and yes all your listed versions except windows (2008) have an 4.3.3 version, i dont know why mongodb dosnt have this, but i hope that the fix makes it more clear on what the error is / was

[helt]

I hope so too, for the other to stumble upon. Would be awesome if the error message contained the url which produced the problem