Resource is not detected by container.init_resources() when declaration is not at class root level

[approxit]

Howdy, another somewhat similar issue like #379, this time with providers.Resource.

Consider this example:

from dependency_injector import providers, containers
import asyncio


class TestObject():
	def __init__(self, foo, bar):
		self.foo = foo
		self.bar = bar


def sync_foo(foo):
	print('sync foo triggered!')
	return foo


async def async_foo(foo):
	print('async foo triggered!')
	await asyncio.sleep(0)
	return foo


class RootResourceContainer(containers.DeclarativeContainer):
	sync_foo_resource=providers.Resource(
		sync_foo,
		foo='bar'
	)
	async_foo_resource=providers.Resource(
		async_foo,
		foo='bar'
	)

	obj_factory = providers.DelegatedFactory(
		TestObject,
		foo=async_foo_resource,
		bar=sync_foo_resource,
	)


class NonRootResourceContainer(containers.DeclarativeContainer):
	obj_factory = providers.DelegatedFactory(
		TestObject,
		foo=providers.Resource(
			async_foo,
			foo='bar'
		),
		bar=providers.Resource(
			sync_foo,
			foo='bar'
		)
	)


async def main():
	container = RootResourceContainer()
	await container.init_resources()
	print('after init')
	print(container.obj_factory())
	await container.shutdown_resources()

	print('----')

	container = NonRootResourceContainer()
	await container.init_resources()  # Crashing as async non-root async Resource is not detected
	print('after init')
	print(container.obj_factory())
	await container.shutdown_resources()


asyncio.run(main())

First container with everything defined at class root level works, as both Resources are detected and initialized by container.init_resources().
Second container with nested Resources definition can't figure it out, and container.init_resources() is not picking up all resources as stated in docs. In result, if no async resources are found, container.init_resources() will not be awaitable, and awaiting it will crash.

Also in first container by calling container.obj_factory() we are receiving awaitable, not actual factory result. As I'm expecting to receive awaitable when there was no previous async Resource initialization and handle this somewhere in my code (as stated in docs), that I'm expecting container.init_resources() to do the job and resolve every declared resource and get rid of awaiting anything inside my code. Or in another words - I'd like to keep Resource-specific logic at the container layer, without bleeding it into my code, which will be sensitive for IoC layer changes.

[rmk135]

Hey @approxit ,

Few things:

1. I'll check if NonRootResourceContainer.init_resources() could check for non root resources as well. That should be possible with some changes done to the code. I don't feel like it should break anything, but I need to double-check. Generally, I'm thinking positively about getting it implemented.

2. I'm not sure if I got what you're saying correct, so let me provide an example. Imagine you create a singleton async function like this:

import asyncio


storage = None


async def get_async_singleton():
    global storage
    if storage is None:
        await asyncio.sleep(0.1)
        storage = object()
    return storage


async def main():
    object1 = await get_async_singleton()
    object2 = await get_async_singleton()
    assert object1 is object2
    ...


if __name__ == '__main__':
    asyncio.run(main())

Function get_async_singleton() returns awaitable object when called second time. This happens even despite second call does not use await inside. This follows the paradigm that "async is always async". It is needed to make sure you can safely await for an async function. The same paradigm is implemented for providers working in async mode. Initializing an async resource does not make it to be sync afterwards. It will keep returning awaitable even despite there is nothing to await anymore. This is an important part of the design. Does it make any sense to you?

[approxit]

🤔

Okay, it looks like I've stretch async Resources understanding way too much, and assumed that their asyncness can be somewhat "transparent" as everything besides providers.Coroutine is sync. So my wrongly assumed code example:

from dependency_injector import providers, containers
import asyncio


async def async_connect(resource):
	await resource.connect()
	yield resource
	await resource.disconnect()


class StatefullClient:
	async def connect(self):
		pass

	async def disconnect(self):
		pass


class SomeService:
	def __init__(self, client):
		self._client = client

	def hello_world(self):
		print('Hello world!')


class SomeContainer(containers.DeclarativeContainer):
	# lets just force async init_resources by line below
	other_client = providers.Resource(
		async_connect,
		providers.Factory(
			StatefullClient
		)
	)

	some_service = providers.Singleton(
		SomeService,
		client=providers.Resource(
			async_connect,
			providers.Factory(
				StatefullClient
			)
		)
	)


async def main():
	container = SomeContainer()
	await container.init_resources()
	service = container.some_service()
	service.hello_world()  # Awaitable instead of actual service here!
	await container.shutdown_resources()


asyncio.run(main())

Should be modified to keep connection/resource handling away from sync providers stack as example below?

from dependency_injector import providers, containers
import asyncio


async def async_connect(resource):
	await resource.connect()
	yield resource
	await resource.disconnect()


class StatefullClient:
	async def connect(self):
		pass

	async def disconnect(self):
		pass


class SomeService:
	def __init__(self, client):
		self._client = client

	def hello_world(self):
		print('Hello world!')


class SomeContainer(containers.DeclarativeContainer):
	other_client = providers.Resource(
		async_connect,
		providers.Factory(
			StatefullClient
		)
	)

	some_service_client = providers.Factory(
		StatefullClient
	)

	some_service_client_connection = providers.Resource(
		async_connect,
		some_service_client
	)

	some_service = providers.Singleton(
		SomeService,
		client=some_service_client
	)


async def main():
	container = SomeContainer()
	await container.init_resources()
	service = container.some_service()
	service.hello_world()  # Actual service here!
	await container.shutdown_resources()


asyncio.run(main())

I get it now, but I'm not gonna lie, my wrong assumption had quite appealing syntax to keep things packed. With "all declaration level" resource initialization it would work so nice... 😄

[rmk135]

I had the same vision on async resources when started working on the feature. I thought about Resource provider as a connector between async and sync worlds. While I proceed with implementation and testing, I realized that it leads to undesired side effects. To avoid the side effects I decided to make it work truly async on all levels.

The goods news is that truly async design brings outstanding feature: framework can collect async dependencies asynchronously. For instance next code will block 3 times:

object = SomeClass(
    resource1=await resource1(),
    resource2=await resource2(),
    resource3=await resource3(),
)

Resource 2 will start initialization only after resource 1 is ready, resource 3 - only after 2 is ready.

Dependency Injector prepares all dependencies concurrently instead:

resource1, resource2, resource3 = await asyncio.gather(resource1(), resource2(), resource3())

object = SomeClass(
    resource1=resource1,
    resource2=resource2,
    resource3=resource3,
)

Resource 1,2,3 are initialized at the same time. Injections are done when the last is ready.

---

As of your example, there are 2 things.

1. You need to await container.some_service. It's needed because container.some_service has async dependency that turns it into async mode.

async def main():
	container = SomeContainer()
	await container.init_resources()
	service = await container.some_service()  # <-- Add await here
	service.hello_world()  # Actual service here!
	await container.shutdown_resources()

2. Keep resources list flat to have await container.init_resources() and await container.shutdown_resources() working properly. This is a temporary measure. I'll fix it.

[rmk135]

Hey @approxit ,

I have published version 4.14.0 that fixes non-root resources initialization bug. The output of the initial example now looks like this:

async def main():
    container = RootResourceContainer()
    await container.init_resources()
    print('after init')
    print(await container.obj_factory())
    await container.shutdown_resources()

    print('----')

    container = NonRootResourceContainer()
    await container.init_resources()  # Is not crashing anymore
    print('after init')
    print(await container.obj_factory())
    await container.shutdown_resources()

    # Output:
    #
    # sync foo triggered!
    # async foo triggered!
    # after init
    # <__main__.TestObject object at 0x106564fa0>
    # ----
    # sync foo triggered!
    # async foo triggered!
    # after init
    # <__main__.TestObject object at 0x106564fa0>

[rmk135]

Hey @approxit , I think I mislead you with my pre-last response and I'm sorry about that.

So few things:

---

1. In my pre-last response I have suggested to add an await in a place that did not require this. It happened because I didn't analyze the structure of example container well.

Below is an example of the container that requires await for retrieving the service:

import asyncio

from dependency_injector import providers, containers


async def async_connect(resource):
    await resource.connect()
    yield resource
    await resource.disconnect()


class StatefullClient:
    async def connect(self):
        pass

    async def disconnect(self):
        pass


class SomeService:
    def __init__(self, client):
        self._client = client

    def hello_world(self):
        print('Hello world!')


class SomeContainer(containers.DeclarativeContainer):

    # Flat version

    client = providers.Factory(StatefullClient)

    resource = providers.Resource(
        async_connect,
        resource=client,
    )

    some_service1 = providers.Singleton(
        SomeService,
        client=resource,
    )

    # Nested version

    some_service2 = providers.Singleton(
        SomeService,
        client=providers.Resource(
            async_connect,
            resource=providers.Factory(
                StatefullClient
            ),
        ),
    )


async def main():
    container = SomeContainer()
    await container.init_resources()

    service1 = await container.some_service1()
    service1.hello_world()  # Actual service here!

    service2 = await container.some_service2()
    service2.hello_world()  # Actual service here!

    await container.shutdown_resources()


asyncio.run(main())

---

2. I found a workaround how to achieve resources synchronization. The magic part is resource.disable_async_mode(). Here is what happens:

• All resources are initialized asynchronously with await container.init_resources().
• After that we disable async mode for the resources
• Resources provide the initialized instanced on step 1 and don't go into async mode anymore

import asyncio

from dependency_injector import providers, containers


async def async_connect(resource):
    await resource.connect()
    yield resource
    await resource.disconnect()


class StatefullClient:
    async def connect(self):
        pass

    async def disconnect(self):
        pass


class SomeService:
    def __init__(self, client):
        self._client = client

    def hello_world(self):
        print('Hello world!')


class SomeContainer(containers.DeclarativeContainer):

    # Flat version

    client = providers.Factory(StatefullClient)

    resource = providers.Resource(
        async_connect,
        resource=client,
    )

    some_service1 = providers.Singleton(
        SomeService,
        client=resource,
    )

    # Nested version

    some_service2 = providers.Singleton(
        SomeService,
        client=providers.Resource(
            async_connect,
            resource=providers.Factory(
                StatefullClient
            ),
        ),
    )


async def main():
    container = SomeContainer()
    await container.init_resources()

    for resource in container.traverse(types=[providers.Resource]):
        resource.disable_async_mode()  # <-- all resources provide just what was initialized

    service1 = container.some_service1()
    service1.hello_world()  # Actual service here!

    service2 = container.some_service2()
    service2.hello_world()  # Actual service here!

    await container.shutdown_resources()


asyncio.run(main())

---

3. I've got an email with your comment. I can not find it here anymore cause you probably have deleted it. It contained idea about .injected attribute, and I find it nice. I think that what we're looking for here is called "synchronization". A little switch between sync and async worlds. In that case I imagine an attribute .synchronized that will play the role of that switch. Consider next container example:

class SomeContainer(containers.DeclarativeContainer):

    # Flat version

    client = providers.Factory(StatefullClient)

    resource = providers.Resource(
        async_connect,
        resource=client,
    )

    some_service1 = providers.Singleton(
        SomeService,
        client=resource.synchronized,  # <-- inject async resource synchronously
    )

    # Nested version

    some_service2 = providers.Singleton(
        SomeService,
        client=providers.Synchronized(  # <-- inject async resource synchronously
            providers.Resource(
                async_connect,
                resource=providers.Factory(
                    StatefullClient
                ),
            ),
        ),
    )

PS: This doesn't work yet.

---

Appreciate your feedback.

[approxit]

Don't worry @rmk135, there was no misleading in your answers. 😄

---

1: Yup, if we want to access any provider, which have async resource in its deps, outside of container, we need await it - got that from your first comments. 😄
Actually worth point out is interacting with container from outside and from inside. All previous examples shows case where we want interact with container from outside. Another option for getting rid of awaiting SomeService is by moving hello_world call to providers.Callable, and voila!

from dependency_injector import providers, containers
import asyncio


async def async_connect(resource):
	print('Connecting resource!')
	await resource.connect()
	yield resource
	await resource.disconnect()


class StatefullClient:
	async def connect(self):
		pass

	async def disconnect(self):
		pass


class SomeService:
	def __init__(self, client):
		self._client = client

	def hello_world(self):
		print('Hello world!')

def container_main(some_service):
	print('Calling service!')
	some_service.hello_world()


class SomeContainer(containers.DeclarativeContainer):
	other_client=providers.Resource(
		async_connect,
		providers.Factory(
			StatefullClient
		)
	)

	some_service = providers.Singleton(
		SomeService,
		client=providers.Resource(
			async_connect,
			providers.Factory(
				StatefullClient
			)
		)
	)

	main = providers.Callable(
		container_main,
		some_service=some_service,
	)


async def main():
	container = SomeContainer()
	await container.init_resources()
	container.main()
	await container.shutdown_resources()


asyncio.run(main())

Either way, all of these solutions fails with providers.Delegated* stuff. Making something delegated is like interacting outside of container - you need handle awaits. Using DelegatedFactory is great for creating per-request handlers. If that handlers will differ in deps, we have bunch of mixed sync and async factories. I'd love to get rid of that awaits when my service is receiving collection of handler factories. Currently using inspect.isawaitable is only reasonable way to go for me...

---

2: That's quite neat idea! So you can force async resource to sync after all... 🤔

---

3: I knew that you will mention that deleted comment... 😂 It was tossed to garbage bin, because it was rant about something that in the end just works with dependency_injector: SomeService is not receiving awaitable in its constructor. SomeService is receiving already resolved async Resource. We have differences of playing with dependency_injector outside of container and inside of container. We need to handle asyncs from outside, but inside deps are resolved and passed as sync. After realization and face palm, I just removed it.

...but taking the .synchronized looks very promising! :)

Of course when we need somehow handle case when synchronized async Resource is not yet initialized but already accessed.
As .synchronized would be great - how providers.Synchronized would behave? It would be a somewhat transparent pass-through provider to mark resource directly given in argument? How about multiple arguments? Or would it mark all resources in dependency tree? How about case when there is no resource in deps tree? It would raise, or stay silent? There are some questions that needs to be answered to prepare implementation and needs bunch of docs to explain all these nuances to the user.
How about keeping it simple and besides .synchronized, introduce just... providers.SynchronizedResource? In my case I would use this variant everywhere, because it suits me more. If would just inherit from providers.Resource and... that's all. there is no need for vast explanation besides mentioning "auto .synchronized".

And it's just me, or we just came to implementation of our original idea? 😅

[rmk135]

Example with callable is terrific:

async def main():
	container = SomeContainer()
	await container.init_resources()
	container.main()
	await container.shutdown_resources()

I have no idea why that works :) I mean why container.main() works without await. I have feeling I need to fix something to make it stop working :)

As of the .synchronized. It's just an idea for now. I will experiment if it's possible to make it work properly.

Despite .synchronized experiment, I would recommend to check if you could go async on all the layers of your application. Python works that way that with a first async / await you turn your application into asynchronous mode. Dependency Injector async mode feature is designed the way to handle this properly: not synchronize, but pass through. If your application already handles asynchronous call on the top level (main function, endpoints etc) then you could turn all providers in async mode explicitly:

    for resource in container.traverse():
        resource.enable_async_mode()

In that case you will need to always use await to get dependencies from the container.

If you have any other thoughts or idea - please, share. Feedback helps to improve the framework.