Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Singletons are created twice when observers subsystem is used #2673

Closed
sarxos opened this issue May 31, 2019 · 8 comments
Closed

Singletons are created twice when observers subsystem is used #2673

sarxos opened this issue May 31, 2019 · 8 comments
Assignees
Labels
area/arc Issue related to ARC (dependency injection) kind/bug Something isn't working
Milestone

Comments

@sarxos
Copy link
Contributor

sarxos commented May 31, 2019

Describe the bug

Singletons by definition should be created only once, however, in the scenario I describe in the next section I managed to make Quarkus create singleton beans twice.

Expected behavior

Singletons are singletons (single instances of a bean).

Actual behavior

Singletons are twinletons (two instances of the same bean).

To Reproduce

Have a non-managed bean:

public class MagicSystem {

	public static volatile int instances = 0;

	public MagicSystem() {
		if (++instances > 1) {
			throw new IllegalStateException("I should be singleton, dude!");
		}
	}
}

Have another bean, a @Singleton with a producer method:

@Singleton
public class MagicSystemFactory {

	public MagicSystemFactory() {
		new Error().printStackTrace();
	}

	@Produces
	@Singleton
	public MagicSystem create() {
		return new MagicSystem();
	}
}

And have observer bean with observer method:

@Singleton
public class MagicObserver {

	private final MagicSystem system;

	@Inject
	public MagicObserver(final MagicSystem system) {
		this.system = system;
	}

	public void observe(@Observes Object event) {
		System.out.println("observed " + event);
	}
}

And a very simple test just to bootstrap Quarkus:

@QuarkusTest
public class MagicTest {

	@Test
	void test_isTrueReallyTrue() {
		assertThat(true).isTrue();
	}
}

After test is run you will observe IllegalStateException with message "I should be singleton, dude!" which indicates that bean was created more than once which is not expected since all these beans are singletons and as such should be a single instances.

13:18:04.417 [main] ERROR io.quarkus.runtime.StartupContext - Running a shutdown task failed
java.lang.IllegalStateException: I should be singleton, dude!
	at com.github.sarxos.abberwoult.cdi.arc2.MagicSystem.<init>(MagicSystem.java:9)
	at com.github.sarxos.abberwoult.cdi.arc2.MagicSystemFactory.create(MagicSystemFactory.java:17)
	at com.github.sarxos.abberwoult.cdi.arc2.MagicSystemFactory_ProducerMethod_create_e2a233282ae40a1503f725ebdfb96bbfe5dd852c_Bean.create(Unknown Source)
	at com.github.sarxos.abberwoult.cdi.arc2.MagicSystemFactory_ProducerMethod_create_e2a233282ae40a1503f725ebdfb96bbfe5dd852c_Bean.create(Unknown Source)
	at io.quarkus.arc.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:86)
	at io.quarkus.arc.AbstractSharedContext.lambda$new$0(AbstractSharedContext.java:33)
	at io.quarkus.arc.ComputingCache$CacheFunction.lambda$apply$0(ComputingCache.java:115)
	at io.quarkus.arc.LazyValue.get(LazyValue.java:42)
	at io.quarkus.arc.ComputingCache.getValue(ComputingCache.java:57)
	at io.quarkus.arc.AbstractSharedContext.get(AbstractSharedContext.java:39)
	at com.github.sarxos.abberwoult.cdi.arc2.MagicSystemFactory_ProducerMethod_create_e2a233282ae40a1503f725ebdfb96bbfe5dd852c_Bean.get(Unknown Source)
	at com.github.sarxos.abberwoult.cdi.arc2.MagicSystemFactory_ProducerMethod_create_e2a233282ae40a1503f725ebdfb96bbfe5dd852c_Bean.get(Unknown Source)
	at com.github.sarxos.abberwoult.cdi.arc2.MagicObserver_Bean.create(Unknown Source)
	at com.github.sarxos.abberwoult.cdi.arc2.MagicObserver_Bean.create(Unknown Source)
	at io.quarkus.arc.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:86)
	at io.quarkus.arc.AbstractSharedContext.lambda$new$0(AbstractSharedContext.java:33)
	at io.quarkus.arc.ComputingCache$CacheFunction.lambda$apply$0(ComputingCache.java:115)
	at io.quarkus.arc.LazyValue.get(LazyValue.java:42)
	at io.quarkus.arc.ComputingCache.getValue(ComputingCache.java:57)
	at io.quarkus.arc.AbstractSharedContext.get(AbstractSharedContext.java:39)
	at com.github.sarxos.abberwoult.cdi.arc2.MagicObserver_Bean.get(Unknown Source)
	at com.github.sarxos.abberwoult.cdi.arc2.MagicObserver_Bean.get(Unknown Source)
	at com.github.sarxos.abberwoult.cdi.arc2.MagicObserver_Observer_observe_e59a6f556572ff5acee13086fbd6a12e1fcef17d.notify(Unknown Source)
	at io.quarkus.arc.EventImpl$Notifier.notify(EventImpl.java:244)
	at io.quarkus.arc.EventImpl$Notifier.notify(EventImpl.java:234)
	at io.quarkus.arc.RequestContext.fireIfNotEmpty(RequestContext.java:120)
	at io.quarkus.arc.RequestContext.destroy(RequestContext.java:151)
	at io.quarkus.arc.ManagedContext.terminate(ManagedContext.java:52)
	at io.quarkus.arc.ArcContainerImpl.shutdown(ArcContainerImpl.java:285)
	at io.quarkus.arc.Arc.shutdown(Arc.java:53)
	at io.quarkus.arc.runtime.ArcDeploymentTemplate$1.run(ArcDeploymentTemplate.java:53)
	at io.quarkus.runtime.StartupContext.close(StartupContext.java:59)
	at io.quarkus.runner.ApplicationImpl1.doStop(Unknown Source)
	at io.quarkus.runtime.Application.stop(Application.java:167)
	at io.quarkus.runner.RuntimeRunner$1.close(RuntimeRunner.java:136)
	at io.quarkus.runner.RuntimeRunner.close(RuntimeRunner.java:83)
	at io.quarkus.test.junit.QuarkusTestExtension$5.close(QuarkusTestExtension.java:242)
	at io.quarkus.test.junit.QuarkusTestExtension$ExtensionState.close(QuarkusTestExtension.java:361)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.jupiter.engine.execution.ExtensionValuesStore.closeAllStoredCloseableValues(ExtensionValuesStore.java:61)

The new Error().printStackTrace() in MagicSystemFactory helps us narrow down the problem and we can see that first singletons are created on test initialization and the second ones are created when test is stopped.

java.lang.Error
	at com.github.sarxos.abberwoult.cdi.arc2.MagicSystemFactory.<init>(MagicSystemFactory.java:11)
	at com.github.sarxos.abberwoult.cdi.arc2.MagicSystemFactory_Bean.create(Unknown Source)
	at com.github.sarxos.abberwoult.cdi.arc2.MagicSystemFactory_Bean.create(Unknown Source)
	at io.quarkus.arc.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:86)
	at io.quarkus.arc.AbstractSharedContext.lambda$new$0(AbstractSharedContext.java:33)
	at io.quarkus.arc.ComputingCache$CacheFunction.lambda$apply$0(ComputingCache.java:115)
	at io.quarkus.arc.LazyValue.get(LazyValue.java:42)
	at io.quarkus.arc.ComputingCache.getValue(ComputingCache.java:57)
	at io.quarkus.arc.AbstractSharedContext.get(AbstractSharedContext.java:39)
	at com.github.sarxos.abberwoult.cdi.arc2.MagicSystemFactory_Bean.get(Unknown Source)
	at com.github.sarxos.abberwoult.cdi.arc2.MagicSystemFactory_Bean.get(Unknown Source)
	at com.github.sarxos.abberwoult.cdi.arc2.MagicSystemFactory_ProducerMethod_create_e2a233282ae40a1503f725ebdfb96bbfe5dd852c_Bean.create(Unknown Source)
	at com.github.sarxos.abberwoult.cdi.arc2.MagicSystemFactory_ProducerMethod_create_e2a233282ae40a1503f725ebdfb96bbfe5dd852c_Bean.create(Unknown Source)
	at io.quarkus.arc.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:86)
	at io.quarkus.arc.AbstractSharedContext.lambda$new$0(AbstractSharedContext.java:33)
	at io.quarkus.arc.ComputingCache$CacheFunction.lambda$apply$0(ComputingCache.java:115)
	at io.quarkus.arc.LazyValue.get(LazyValue.java:42)
	at io.quarkus.arc.ComputingCache.getValue(ComputingCache.java:57)
	at io.quarkus.arc.AbstractSharedContext.get(AbstractSharedContext.java:39)
	at com.github.sarxos.abberwoult.cdi.arc2.MagicSystemFactory_ProducerMethod_create_e2a233282ae40a1503f725ebdfb96bbfe5dd852c_Bean.get(Unknown Source)
	at com.github.sarxos.abberwoult.cdi.arc2.MagicSystemFactory_ProducerMethod_create_e2a233282ae40a1503f725ebdfb96bbfe5dd852c_Bean.get(Unknown Source)
	at com.github.sarxos.abberwoult.cdi.arc2.MagicObserver_Bean.create(Unknown Source)
	at com.github.sarxos.abberwoult.cdi.arc2.MagicObserver_Bean.create(Unknown Source)
	at io.quarkus.arc.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:86)
	at io.quarkus.arc.AbstractSharedContext.lambda$new$0(AbstractSharedContext.java:33)
	at io.quarkus.arc.ComputingCache$CacheFunction.lambda$apply$0(ComputingCache.java:115)
	at io.quarkus.arc.LazyValue.get(LazyValue.java:42)
	at io.quarkus.arc.ComputingCache.getValue(ComputingCache.java:57)
	at io.quarkus.arc.AbstractSharedContext.get(AbstractSharedContext.java:39)
	at com.github.sarxos.abberwoult.cdi.arc2.MagicObserver_Bean.get(Unknown Source)
	at com.github.sarxos.abberwoult.cdi.arc2.MagicObserver_Bean.get(Unknown Source)
	at com.github.sarxos.abberwoult.cdi.arc2.MagicObserver_Observer_observe_e59a6f556572ff5acee13086fbd6a12e1fcef17d.notify(Unknown Source)
	at io.quarkus.arc.EventImpl$Notifier.notify(EventImpl.java:244)
	at io.quarkus.arc.EventImpl$Notifier.notify(EventImpl.java:234)
	at io.quarkus.arc.ArcContainerImpl.init(ArcContainerImpl.java:143)
	at io.quarkus.arc.Arc.initialize(Arc.java:36)
	at io.quarkus.arc.runtime.ArcDeploymentTemplate.getContainer(ArcDeploymentTemplate.java:49)
	at io.quarkus.deployment.steps.ArcAnnotationProcessor$build6.deploy_0(Unknown Source)
	at io.quarkus.deployment.steps.ArcAnnotationProcessor$build6.deploy(Unknown Source)
	at io.quarkus.runner.ApplicationImpl1.<clinit>(Unknown Source)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at java.lang.Class.newInstance(Class.java:442)
	at io.quarkus.runner.RuntimeRunner.run(RuntimeRunner.java:127)
	at io.quarkus.test.junit.QuarkusTestExtension.doJavaStart(QuarkusTestExtension.java:237)
	at io.quarkus.test.junit.QuarkusTestExtension.createTestInstance(QuarkusTestExtension.java:303)
	at org.junit.jupiter.engine.descriptor.ClassTestDescriptor.invokeTestInstanceFactory(ClassTestDescriptor.java:299)
	at org.junit.jupiter.engine.descriptor.ClassTestDescriptor.instantiateTestClass(ClassTestDescriptor.java:289)
java.lang.Error
	at com.github.sarxos.abberwoult.cdi.arc2.MagicSystemFactory.<init>(MagicSystemFactory.java:11)
	at com.github.sarxos.abberwoult.cdi.arc2.MagicSystemFactory_Bean.create(Unknown Source)
	at com.github.sarxos.abberwoult.cdi.arc2.MagicSystemFactory_Bean.create(Unknown Source)
	at io.quarkus.arc.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:86)
	at io.quarkus.arc.AbstractSharedContext.lambda$new$0(AbstractSharedContext.java:33)
	at io.quarkus.arc.ComputingCache$CacheFunction.lambda$apply$0(ComputingCache.java:115)
	at io.quarkus.arc.LazyValue.get(LazyValue.java:42)
	at io.quarkus.arc.ComputingCache.getValue(ComputingCache.java:57)
	at io.quarkus.arc.AbstractSharedContext.get(AbstractSharedContext.java:39)
	at com.github.sarxos.abberwoult.cdi.arc2.MagicSystemFactory_Bean.get(Unknown Source)
	at com.github.sarxos.abberwoult.cdi.arc2.MagicSystemFactory_Bean.get(Unknown Source)
	at com.github.sarxos.abberwoult.cdi.arc2.MagicSystemFactory_ProducerMethod_create_e2a233282ae40a1503f725ebdfb96bbfe5dd852c_Bean.create(Unknown Source)
	at com.github.sarxos.abberwoult.cdi.arc2.MagicSystemFactory_ProducerMethod_create_e2a233282ae40a1503f725ebdfb96bbfe5dd852c_Bean.create(Unknown Source)
	at io.quarkus.arc.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:86)
	at io.quarkus.arc.AbstractSharedContext.lambda$new$0(AbstractSharedContext.java:33)
	at io.quarkus.arc.ComputingCache$CacheFunction.lambda$apply$0(ComputingCache.java:115)
	at io.quarkus.arc.LazyValue.get(LazyValue.java:42)
	at io.quarkus.arc.ComputingCache.getValue(ComputingCache.java:57)
	at io.quarkus.arc.AbstractSharedContext.get(AbstractSharedContext.java:39)
	at com.github.sarxos.abberwoult.cdi.arc2.MagicSystemFactory_ProducerMethod_create_e2a233282ae40a1503f725ebdfb96bbfe5dd852c_Bean.get(Unknown Source)
	at com.github.sarxos.abberwoult.cdi.arc2.MagicSystemFactory_ProducerMethod_create_e2a233282ae40a1503f725ebdfb96bbfe5dd852c_Bean.get(Unknown Source)
	at com.github.sarxos.abberwoult.cdi.arc2.MagicObserver_Bean.create(Unknown Source)
	at com.github.sarxos.abberwoult.cdi.arc2.MagicObserver_Bean.create(Unknown Source)
	at io.quarkus.arc.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:86)
	at io.quarkus.arc.AbstractSharedContext.lambda$new$0(AbstractSharedContext.java:33)
	at io.quarkus.arc.ComputingCache$CacheFunction.lambda$apply$0(ComputingCache.java:115)
	at io.quarkus.arc.LazyValue.get(LazyValue.java:42)
	at io.quarkus.arc.ComputingCache.getValue(ComputingCache.java:57)
	at io.quarkus.arc.AbstractSharedContext.get(AbstractSharedContext.java:39)
	at com.github.sarxos.abberwoult.cdi.arc2.MagicObserver_Bean.get(Unknown Source)
	at com.github.sarxos.abberwoult.cdi.arc2.MagicObserver_Bean.get(Unknown Source)
	at com.github.sarxos.abberwoult.cdi.arc2.MagicObserver_Observer_observe_e59a6f556572ff5acee13086fbd6a12e1fcef17d.notify(Unknown Source)
	at io.quarkus.arc.EventImpl$Notifier.notify(EventImpl.java:244)
	at io.quarkus.arc.EventImpl$Notifier.notify(EventImpl.java:234)
	at io.quarkus.arc.RequestContext.fireIfNotEmpty(RequestContext.java:120)
	at io.quarkus.arc.RequestContext.destroy(RequestContext.java:151)
	at io.quarkus.arc.ManagedContext.terminate(ManagedContext.java:52)
	at io.quarkus.arc.ArcContainerImpl.shutdown(ArcContainerImpl.java:285)
	at io.quarkus.arc.Arc.shutdown(Arc.java:53)
	at io.quarkus.arc.runtime.ArcDeploymentTemplate$1.run(ArcDeploymentTemplate.java:53)
	at io.quarkus.runtime.StartupContext.close(StartupContext.java:59)
	at io.quarkus.runner.ApplicationImpl1.doStop(Unknown Source)
	at io.quarkus.runtime.Application.stop(Application.java:167)
	at io.quarkus.runner.RuntimeRunner$1.close(RuntimeRunner.java:136)
	at io.quarkus.runner.RuntimeRunner.close(RuntimeRunner.java:83)
	at io.quarkus.test.junit.QuarkusTestExtension$5.close(QuarkusTestExtension.java:242)
	at io.quarkus.test.junit.QuarkusTestExtension$ExtensionState.close(QuarkusTestExtension.java:361)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.jupiter.engine.execution.ExtensionValuesStore.closeAllStoredCloseableValues(ExtensionValuesStore.java:61)

When observer method is commented out, the code works as expected!

@Singleton
public class MagicObserver {

	private final MagicSystem system;

	@Inject
	public MagicObserver(final MagicSystem system) {
		this.system = system;
	}

	// COMMENTING THIS CAUSE QUARKUS TO WORK AS EXPECTED
	// public void observe(@Observes Object event) {
	// 	System.out.println("observed " + event);
	// }
}

Configuration

N/A

Screenshots

N/A/

Environment (please complete the following information):

$ uname -a
Linux sarxos-comp 4.4.0-148-generic #174-Ubuntu SMP Tue May 7 12:20:14 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
$ java -version
java version "1.8.0_201"
Java(TM) SE Runtime Environment (build 1.8.0_201-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode)
$ ~/bin/graalvm/bin/java -version
openjdk version "1.8.0_202"
OpenJDK Runtime Environment (build 1.8.0_202-20190206132807.buildslave.jdk8u-src-tar--b08)
OpenJDK GraalVM CE 1.0.0-rc14 (build 25.202-b08-jvmci-0.56, mixed mode)

Quarkus revision:

commit 96f6e88c44777ce68353dd3968eb49186b7ca24d
Merge: dbbdde6 698112c
Author: Stuart Douglas <stuart.w.douglas@gmail.com>
Date:   Fri May 31 13:41:16 2019 +1000

    Merge pull request #2665 from stuartwdouglas/zk-version
    
    Upgrade zookeeper to 3.4.14

Additional context

Found when playing with my project available here https://github.com/sarxos/abberwoult

@sarxos sarxos added the kind/bug Something isn't working label May 31, 2019
@gsmet gsmet added the area/arc Issue related to ARC (dependency injection) label May 31, 2019
@mkouba mkouba self-assigned this May 31, 2019
@mkouba
Copy link
Contributor

mkouba commented May 31, 2019

Ok. So the problem is that MagicObserver.observe() should observe any event, including all @BeforeDestroyed and @Destroyed events fired for all built-in contexts. However, during container shutdown when app and singleton contexts are destroyed Arc also attempts to terminate the request context after the singleton context is destroyed. As a result, MagicObserver is instantiated just to receive the notification (hence the second instance created).

The ordering of context destruction is not defined so even if we try to avoid this particular use case, another problem could be with an @ApplicationScoped observer. I think that this is more a spec issue as there are no requirements for observers of @BeforeDestroyed and @Destroyed events.

It should be save to use @Dependent observers - these are destroyed when the notification completes.

Anyway, I think it would make sense to modify the AbstractSharedContextso that it's not always active and also ArcContainerImpl.resolveObservers() so that it skips observers declared on a bean from a context that is not active. @manovotn WDYT?

@mkouba
Copy link
Contributor

mkouba commented May 31, 2019

Anyway, I think it would make sense to modify the AbstractSharedContextso that it's not always active and also ArcContainerImpl.resolveObservers() so that it skips observers declared on a bean from a context that is not active.

Actually, it's probably not a good idea. We should only do this if Reception.IF_EXISTS is used.

@sarxos
Copy link
Contributor Author

sarxos commented May 31, 2019

Thank you @mkouba for a detailed explanation.

Can I workaround this somehow? Some workaround I can think of would be to simply ignore this scenario and skip the event since it's useless for me at this stage (since whole stack is already terminated).

Can I do something like:

public class MagicObserver {

	private final MagicSystem system;

	public MagicObserver() {
		if (isScopeActive()) {
			this.system = CDI.current().select(MagicSystem.class).get();
		} else {
			this.system = null;
		}
	}

	private boolean isScopeActive() {
		return // can I check if scope is active or destroyed here?
	}

	public void observe(@Observes Object event) {
		if (system != null) {
			System.out.println("observed " + event + " " + system);
		}
	}
}

The other w/a (but the ugliest one I can think of) would be to somehow destroy my non-managed bean. When I worked with HK2 I was able to have factory with producer and destroyer methods, e.g:

@Service
public class SomeSystemFactory implements Factory<SomeSystem> {

	@Override
	@Singleton
	public SomeSystem provide() {
		return SomeSystem.create(name, config.get());
	}

	@Override
	public void dispose(SomeSystem system) {
		system.terminate();
	}
}

By doing this I could destroy my bean in a clear way (i.e. close ports, sockets, etc) and, if there is no other option, let it be created again, even just for a moment.

@mkouba
Copy link
Contributor

mkouba commented May 31, 2019

You can also use disposer methods for producers in CDI. See for example https://github.com/quarkusio/quarkus/blob/master/independent-projects/arc/tests/src/test/java/io/quarkus/arc/test/producer/disposer/DisposerTest.java#L89-L92. For isScopeActive() you can either use BeanManager.getContext() or Arc-specific ArcContainer.getActiveContext().

WRT workaround: 1) use @Dependent for your observer, 2) do not observe Object but some more concrete type, 3) use @Observes(notifyObserver = Reception.IF_EXISTS), well, this does not work yet, I need to send a PR ;-)

@sarxos
Copy link
Contributor Author

sarxos commented May 31, 2019

Thank you @mkouba!

I already tried using @Dependent for my observer, but this didn't help.

@Dependent
public class MagicObserver {

	private final MagicSystem system;

	@Inject
	public MagicObserver(final MagicSystem system) {
		this.system = system;
	}

	public void observe(@Observes Object event) {
		System.out.println("observed " + event);
	}
}

The problem in fact is not with the multiple instances of MagicObserver, but with the fact that Quarkus is spawning new instance of MagicSystem. In my case this system has a remoting capabilities and creating cause Java to say "nay, I cannot bind to the already used port".

This is weird because in ArcContainerImpl I see that singleton scope is destroyed only after notification is done...

EventImpl.createNotifier(Object.class, Object.class, destroyQualifiers, this).notify(toString());
singletonContext.destroy();

@mkouba
Copy link
Contributor

mkouba commented May 31, 2019

I already tried using @dependent for my observer, but this didn't help.

Yes, because when MagicObserver is created it's also injected -> new MagicSystem is created as well. And it's created because we destroy request context afterwards. In fact, the request context should probably be inactive but for some reason it may not be and so we attempt to destroy it as well.

@sarxos
Copy link
Contributor Author

sarxos commented May 31, 2019

I confirm that change from mkouba@a2e0dbc fixed described problem.

@sarxos
Copy link
Contributor Author

sarxos commented Jun 4, 2019

Thank you @mkouba :)

@gsmet gsmet added this to the 0.17.0 milestone Jun 6, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/arc Issue related to ARC (dependency injection) kind/bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants