fix(cosmosdb): Add support for the CosmosDB Emulator

[mbenabda]

Adds support for the CosmosDB Emulator container

[alexanderankin]

since we do not count community features towards testcontainers features, i have tagged this PR with feat-community but renamed the PR to "fix" - i have fixed the docs build for you but i am not going to get the module tests to pass, they seemed to not pass on the last run

[alexanderankin]

you can run the tests locally with poetry install -E cosmosdb --with dev && poetry run pytest -s modules/cosmosdb/tests in case it helps you. i was able to validate that the waiting you are doing is in fact working - the log, the wait for url, etc, I suspect the emulator just needs more than two minutes to startup and is timing out? not sure.

[mbenabda]

thank you @alexanderankin for looking into this.

Indeed, the container unfortunately takes a while to start up, and I believe you're correct: this is timeout-related failure.

locally, the test passes, and runs consistently in ~1.46s in avg (avg calculated over 100 successive test runs. all passed)

$ time pytest -s modules/cosmosdb/tests
=========================================================================================== test session starts ============================================================================================
platform linux -- Python 3.10.12, pytest-7.4.3, pluggy-1.4.0
rootdir: /tmp/tmp.olcVZbq8gB/testcontainers-python
configfile: pyproject.toml
plugins: asyncio-0.23.5, anyio-4.3.0, cov-4.1.0
asyncio: mode=strict
collected 1 item                                                                                                                                                                                           

modules/cosmosdb/tests/test_cosmosdb.py::test_docker_run Pulling image testcontainers/ryuk:0.7.0

---------------------------------------------------------------------------------------------- live log call -----------------------------------------------------------------------------------------------
INFO     testcontainers.core.container:container.py:88 Pulling image testcontainers/ryuk:0.7.0
Container started: 958785fad0f5
INFO     testcontainers.core.container:container.py:101 Container started: 958785fad0f5
Waiting for container <Container: 958785fad0f5> with image testcontainers/ryuk:0.7.0 to be ready ...
INFO     testcontainers.core.waiting_utils:waiting_utils.py:52 Waiting for container <Container: 958785fad0f5> with image testcontainers/ryuk:0.7.0 to be ready ...
Pulling image mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest
INFO     testcontainers.core.container:container.py:88 Pulling image mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest
Container started: 2435177c5b4e
INFO     testcontainers.core.container:container.py:101 Container started: 2435177c5b4e
Waiting for container <Container: 2435177c5b4e> with image mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest to be ready ...
INFO     testcontainers.core.waiting_utils:waiting_utils.py:52 Waiting for container <Container: 2435177c5b4e> with image mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest to be ready ...
Waiting for container <Container: 2435177c5b4e> with image mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest to be ready ...
INFO     testcontainers.core.waiting_utils:waiting_utils.py:52 Waiting for container <Container: 2435177c5b4e> with image mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest to be ready ...
Waiting for container <Container: 2435177c5b4e> with image mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest to be ready ...
INFO     testcontainers.core.waiting_utils:waiting_utils.py:52 Waiting for container <Container: 2435177c5b4e> with image mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest to be ready ...
Waiting for container <Container: 2435177c5b4e> with image mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest to be ready ...
INFO     testcontainers.core.waiting_utils:waiting_utils.py:52 Waiting for container <Container: 2435177c5b4e> with image mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest to be ready ...

(...)

================================================================================ 1 passed, 2 warnings in 105.71s (0:01:45) =================================================================================

real	1m45,927s
user	0m0,875s
sys	0m0,079s

[mbenabda]

@alexanderankin could we move this PR back to a draft ? it'll take more time than i expected to have something proper:

• The emulator container doesn't support port mapping (known limitation)

• I'm binding port 8081 to the host to check for readiness of the emulator's endpoints
  => there can't be 2 running containers at any given time

• Tests for != python versions seem to be running on the same host
  => tests are bound to fail

I plan on removing the endpoint readiness check from the module, and document how one would proceed to check for that.

Sorry for the inconvenience this has been ...

[alexanderankin]

no worries, if you can get the tests to pass we can merge as is and make changes later as well.

i noticed the port limitation myself so i wouldn't worry about changing that

if you can change the timeout configuration during tests so that the tests do actually pass - this is probably the most critical thing - as it is demonstrated in #578, #532

[mbenabda]

@alexanderankin Hi, i'd like to try running the tests workflows with this. is it possible ? thank you

[alexanderankin]

looks like "failing after 10m" still taking a while

[mbenabda]

@alexanderankin looks like we're good now:
https://github.com/mbenabda/testcontainers-python/actions/runs/9254875065

I missed the part where embeded code examples were actually executed as part of the CI

I had to skip the examples from this module from being executed
because to have them run successfully in CI, the examples would have to be changed in a way that doesn't reflect the actual module usage (ie: add bind_ports=False to the container constructors parameters)

unrelated question: shall I squash my commits ?

Thanks

[kiview]

What does this mean?

The emulator container doesn't support port mapping (known limitation)

Wondering, since I am not aware of any limitation in this regard from our Java implementation.

[mbenabda]

@kiview

What does this mean?

The emulator container doesn't support port mapping (known limitation)

Wondering, since I am not aware of any limitation in this regard from our Java [implementation]>(https://github.com/testcontainers/testcontainers-java/blob/main/modules/azure/src/main/java/org/testcontainers/containers/CosmosDBEmulatorContainer.java).

Now that you mention it, it sounds like BS.

To be honest I read that on a couple of occasions and did not bother to check 👎
I'll look into it and push changes to remove the static port bindings if that pans out

Thank you for pointing that out

[alexanderankin]

squashing the commits is not necessary. i am really not sure how it works in java because when i was stepping through the python driver, it had some abstractions around determining this port and it was not clear how to inform the azure cosmos driver that it should be using a different port (from the standpoint of my debugging - perhaps if i started from the docs instead it would have been clear)

[eddumelendez]

cosmosdb client can use two modes: direct and gateway. gateway works in java with random ports and direct is not supported. See testcontainers/testcontainers-java#5518

[saikotek]

Hello, when it will be added? I too implemented comsosdb emulator support but @mbenabda did it better.

squashing the commits is not necessary. i am really not sure how it works in java because when i was stepping through the python driver, it had some abstractions around determining this port and it was not clear how to inform the azure cosmos driver that it should be using a different port (from the standpoint of my debugging - perhaps if i started from the docs instead it would have been clear)

let's be realistic, this emulator kinda sucks and is heavily restricted. I had the problem with binding ports other than 8081 not because of emulator, but because of python's cosmosdb client ignoring my port configuration. Also official documentation publishes ports 10250-10255 and they do not even say what they are for nor if they are needed..
https://learn.microsoft.com/en-us/azure/cosmos-db/how-to-develop-emulator?tabs=docker-linux%2Cpython&pivots=api-nosql#start-the-emulator
btw I'd suggest squashing the commits to keep the main branch clean.

[alexanderankin]

should i just clean up and attempt to merge what is in here?

[mbenabda]

@saikotek @alexanderankin Sorry, I didn't find the time to get back to this these past couple of weeks.

@alexanderankin feel free clean up and merge, it should work with the caveat of binding static ports.

I'll get back to this as soon as things calm down on my side (hopefully starting from next week), and if you've merged by then, open another PR to improve on this basis.

[alexanderankin]

present pr - main...a85a9b6

rebased - main...pr/cosmosdb_emulator.tmp
rebased permalink - main...b650634

rebased main - main...pr/cosmosdb_emulator.tmp2
rebased main permalink - main...fd04554

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Please upload report for BASE (main@e575b28). Learn more about missing BASE report.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #579   +/-   ##
=======================================
  Coverage        ?   76.26%           
=======================================
  Files           ?       11           
  Lines           ?      573           
  Branches        ?       83           
=======================================
  Hits            ?      437           
  Misses          ?      110           
  Partials        ?       26           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.