Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

A concurrency issue #190

Open
endison1986 opened this issue Feb 18, 2024 · 5 comments
Open

A concurrency issue #190

endison1986 opened this issue Feb 18, 2024 · 5 comments

Comments

@endison1986
Copy link
Contributor

During development, I got an exception log that looked like it was looping to a null value. So I wrote a piece of test code to try to restore the error. Judging from the test code, this should be a concurrency issue.

Is this result expected?

dominion.SystemScheduler invoke 
java.lang.NullPointerException
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
	at java.base/java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:564)
	at java.base/java.util.concurrent.ForkJoinTask.reportException(ForkJoinTask.java:591)
	at java.base/java.util.concurrent.ForkJoinTask.joinForPoolInvoke(ForkJoinTask.java:1042)
	at java.base/java.util.concurrent.ForkJoinPool.invoke(ForkJoinPool.java:2639)
	at dev.dominion.ecs.engine/dev.dominion.ecs.engine.SystemScheduler.forkAndJoin(SystemScheduler.java:113)
	at dev.dominion.ecs.engine/dev.dominion.ecs.engine.SystemScheduler$Single.call(SystemScheduler.java:280)
	at dev.dominion.ecs.engine/dev.dominion.ecs.engine.SystemScheduler$Single.call(SystemScheduler.java:257)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.NullPointerException: Cannot invoke "dev.dominion.ecs.engine.collections.ChunkedPool$Item.getChunk()" because "item" is null
	at dev.dominion.ecs.engine/dev.dominion.ecs.engine.collections.ChunkedPool$PoolDataIteratorWithState.next(ChunkedPool.java:754)
	at dev.dominion.ecs.engine/dev.dominion.ecs.engine.DataComposition$IteratorWith1Next.next(DataComposition.java:306)
	at dev.dominion.ecs.engine/dev.dominion.ecs.engine.DataComposition$IteratorWith1Next.next(DataComposition.java:294)
	at java.base/java.lang.Iterable.forEach(Iterable.java:74)
	at dev.dominion.ecs.engine/dev.dominion.ecs.engine.Test.lambda$main$0(Test.java:34)
	at dev.dominion.ecs.engine/dev.dominion.ecs.engine.SystemScheduler$2.compute(SystemScheduler.java:116)
	at java.base/java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:194)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)

Test code:

public class Test {
    public enum S {
        A, B
    }
    public record A(int id) {
    }
    public static void consume(Object c) {
        try {
            TimeUnit.MILLISECONDS.sleep(10);
        } catch (InterruptedException e) {
            throw new RuntimeException(e);
        }
    }
    public static void main(String[] args) {
        EntityRepository entityRepository = (EntityRepository) new EntityRepository.Factory().create("test");
        final var list = new ArrayList<Entity>(1000);
        for (int i = 0; i < 1000; i++) {
            list.add(entityRepository.createEntity(new A(i)).setState(S.A));
        }
        Scheduler scheduler = entityRepository.createScheduler();

        scheduler.schedule(() -> entityRepository.findEntitiesWith(A.class).withState(S.A).forEach(Test::consume));


        new Thread(() -> {
            while (true) {
                try {
                    TimeUnit.MILLISECONDS.sleep(10);
                    for (int i = 0; i < 1000; i++) {
                        list.get(i).setState(S.B);
                    }
                } catch (InterruptedException e) {
                    throw new RuntimeException(e);
                }
            }
        }).start();

        scheduler.tickAtFixedRate(20);
    }
}
@endison1986
Copy link
Contributor Author

What do you think about this issue? @enricostara
I think that when performing operations such as addComponent, removeComponent, setState, etc., the Item moved, causing incorrect Items to be traversed when traversing the Chunk.
Can this problem be avoided by avoiding Item movement? However, there will definitely be a certain performance loss due to the existence of Null Item during traversal.

@endison1986
Copy link
Contributor Author

@enricostara Please tell me if you are solving this problem

Solving this problem seems difficult.
I try to avoid moving the last Item to the deleted Item position when the component changes, but retain the original value or NULL value to avoid obtaining uncertain values during traversal, and increase Chunk space recycling and The logic of small chunk merging.
After completing the unit test, the results obtained by AddingComponentBenchmark, CreateEntityBenchmark and DeletingEntityBenchmark are quite satisfactory, but the performance of IteratingComponentBenchmark is not as good as before, roughly tied with Artemis, because when traversing, it will be judged whether the Item is NULL or belongs to the current Chunk.
At present, I have no way to solve the performance problem of this traversal.

@enricostara
Copy link
Contributor

enricostara commented Mar 19, 2024 •
edited

Hi @endison1986,

this code method in CompositionRepository is preventing concurrency issues already for modifying entities (add/remove components) and for the state change as well:

public void modifyComponents(IntEntity entity, PreparedComposition.TargetComposition targetComposition, Object addedComponent, Object[] addedComponents) {
        if (Logging.isLoggable(loggingContext.levelIndex(), System.Logger.Level.DEBUG)) {
            LOGGER.log(
                    System.Logger.Level.DEBUG, Logging.format(loggingContext.subject()
                            , "Modifying " + entity + " from " + entity.getComposition() + " to " + targetComposition.target())
            );
        }
        ChunkedPool.Tenant<IntEntity> prevTenant;
        synchronized (prevTenant = entity.getChunk().getTenant()) {
            int prevId = entity.getId();
            synchronized (targetComposition.target().getTenant()) {
                targetComposition.target().attachEntity(entity, targetComposition.indexMapping(), targetComposition.addedIndexMapping(), addedComponent, addedComponents);
            }
            prevTenant.freeId(prevId);
        }
        if (entity.stateChunk != null) {
            ChunkedPool.Tenant<IntEntity> prevStateTenant;
            synchronized (prevStateTenant = entity.stateChunk.getTenant()) {
                prevStateTenant.freeStateId(entity.getStateId());
                entity.stateChunk = targetComposition.target().fetchStateTenants((IndexKey) prevStateTenant.getSubject()).registerState(entity);
            }
        }
    }

together with the following method in IntEntity:

    @Override
    public synchronized <S extends Enum<S>> Entity setState(S state) {
        if (!isEnabled()) {
            return this;
        }
        if (state == null && stateChunk != null) {
            ChunkedPool.Tenant<IntEntity> tenant;
            synchronized (tenant = stateChunk.getTenant()) {
                tenant.freeStateId(stateId);
                stateChunk = null;
                return this;
            }
        }
        DataComposition composition = getComposition();
        if (stateChunk != null) {
            var tenant = stateChunk.getTenant();
            if (tenant == composition.getStateTenant(state)) {
                return this;
            }
            synchronized (stateChunk.getTenant()) {
                tenant.freeStateId(stateId);
            }
        }
        ChunkedPool.Tenant<IntEntity> tenant;
        synchronized (tenant = composition.fetchStateTenants(state)) {
            stateChunk = tenant.registerState(this);
        }
        return this;
    }

I see concurrent tests are already passing for the add and remove of comps however, there are no proper tests yet for the change of states, even if the synchronised code I see should be fine for this specific case as well.

@endison1986
Copy link
Contributor Author

endison1986 commented Mar 20, 2024 •
edited

Hi, @enricostara
Changing the state will result in an error. Although no error is reported when adding or deleting components, it does not mean that there is no problem.
When withState is traversed, it is because it needs to access the LinkedChunk of two tenants, so the more direct result is an error.
But without withState, the correct entity may not be obtained. In fact, it is easy to understand, because during traversal, the elements of LinkedChunk change (lastItem moveTo removeItem), but traversal does not isolate this change.

In the following example, the final output count is 60 because the first 20 entities created were traversed twice.

         EntityRepository entityRepository = (EntityRepository) new EntityRepository.Factory().create("test");
         for (int i = 0; i < 20; i++) {
             entityRepository.createEntity(new A(i));
         }

         for (int i = 0; i < 20; i++) {
             entityRepository.createEntity(new A(i + 30), new B(i + 30));
         }

         var count = new AtomicInteger(0);
         entityRepository.findEntitiesWith(A.class).stream().forEach(rs -> {
             if(!rs.entity().has(B.class)) {
                 rs.entity().add(new B(rs.comp().id()));
             }
             count.incrementAndGet();
         });

         System.out.println(count);

The result is

With1[comp=A[id=19], entity=Entity={id=|0:2:20|-> [A[id=19], B[id=19]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=18], entity=Entity={id=|0:2:21|-> [A[id=18], B[id=18]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=17], entity=Entity={id=|0:2:22|-> [A[id=17], B[id=17]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=16], entity=Entity={id=|0:2:23|-> [A[id=16], B[id=16]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=15], entity=Entity={id=|0:2:24|-> [A[id=15], B[id=15]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=14], entity=Entity={id=|0:2:25|-> [A[id=14], B[id=14]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=13], entity=Entity={id=|0:2:26|-> [A[id=13], B[id=13]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=12], entity=Entity={id=|0:2:27|-> [A[id=12], B[id=12]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=11], entity=Entity={id=|0:2:28|-> [A[id=11], B[id=11]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=10], entity=Entity={id=|0:2:29|-> [A[id=10], B[id=10]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=9], entity=Entity={id=|0:2:30|-> [A[id=9], B[id=9]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=8], entity=Entity={id=|0:2:31|-> [A[id=8], B[id=8]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=7], entity=Entity={id=|0:2:32|-> [A[id=7], B[id=7]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=6], entity=Entity={id=|0:2:33|-> [A[id=6], B[id=6]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=5], entity=Entity={id=|0:2:34|-> [A[id=5], B[id=5]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=4], entity=Entity={id=|0:2:35|-> [A[id=4], B[id=4]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=3], entity=Entity={id=|0:2:36|-> [A[id=3], B[id=3]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=2], entity=Entity={id=|0:2:37|-> [A[id=2], B[id=2]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=1], entity=Entity={id=|0:2:38|-> [A[id=1], B[id=1]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=0], entity=Entity={id=|0:2:39|-> [A[id=0], B[id=0]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=0], entity=Entity={id=|0:2:39|-> [A[id=0], B[id=0]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=1], entity=Entity={id=|0:2:38|-> [A[id=1], B[id=1]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=2], entity=Entity={id=|0:2:37|-> [A[id=2], B[id=2]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=3], entity=Entity={id=|0:2:36|-> [A[id=3], B[id=3]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=4], entity=Entity={id=|0:2:35|-> [A[id=4], B[id=4]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=5], entity=Entity={id=|0:2:34|-> [A[id=5], B[id=5]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=6], entity=Entity={id=|0:2:33|-> [A[id=6], B[id=6]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=7], entity=Entity={id=|0:2:32|-> [A[id=7], B[id=7]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=8], entity=Entity={id=|0:2:31|-> [A[id=8], B[id=8]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=9], entity=Entity={id=|0:2:30|-> [A[id=9], B[id=9]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=10], entity=Entity={id=|0:2:29|-> [A[id=10], B[id=10]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=11], entity=Entity={id=|0:2:28|-> [A[id=11], B[id=11]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=12], entity=Entity={id=|0:2:27|-> [A[id=12], B[id=12]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=13], entity=Entity={id=|0:2:26|-> [A[id=13], B[id=13]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=14], entity=Entity={id=|0:2:25|-> [A[id=14], B[id=14]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=15], entity=Entity={id=|0:2:24|-> [A[id=15], B[id=15]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=16], entity=Entity={id=|0:2:23|-> [A[id=16], B[id=16]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=17], entity=Entity={id=|0:2:22|-> [A[id=17], B[id=17]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=18], entity=Entity={id=|0:2:21|-> [A[id=18], B[id=18]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=19], entity=Entity={id=|0:2:20|-> [A[id=19], B[id=19]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=49], entity=Entity={id=|0:2:19|-> [A[id=49], B[id=49]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=48], entity=Entity={id=|0:2:18|-> [A[id=48], B[id=48]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=47], entity=Entity={id=|0:2:17|-> [A[id=47], B[id=47]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=46], entity=Entity={id=|0:2:16|-> [A[id=46], B[id=46]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=45], entity=Entity={id=|0:2:15|-> [A[id=45], B[id=45]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=44], entity=Entity={id=|0:2:14|-> [A[id=44], B[id=44]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=43], entity=Entity={id=|0:2:13|-> [A[id=43], B[id=43]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=42], entity=Entity={id=|0:2:12|-> [A[id=42], B[id=42]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=41], entity=Entity={id=|0:2:11|-> [A[id=41], B[id=41]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=40], entity=Entity={id=|0:2:10|-> [A[id=40], B[id=40]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=39], entity=Entity={id=|0:2:9|-> [A[id=39], B[id=39]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=38], entity=Entity={id=|0:2:8|-> [A[id=38], B[id=38]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=37], entity=Entity={id=|0:2:7|-> [A[id=37], B[id=37]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=36], entity=Entity={id=|0:2:6|-> [A[id=36], B[id=36]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=35], entity=Entity={id=|0:2:5|-> [A[id=35], B[id=35]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=34], entity=Entity={id=|0:2:4|-> [A[id=34], B[id=34]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=33], entity=Entity={id=|0:2:3|-> [A[id=33], B[id=33]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=32], entity=Entity={id=|0:2:2|-> [A[id=32], B[id=32]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=31], entity=Entity={id=|0:2:1|-> [A[id=31], B[id=31]], stateId=|1:0:0|, enabled=true}]
With1[comp=A[id=30], entity=Entity={id=|0:2:0|-> [A[id=30], B[id=30]], stateId=|1:0:0|, enabled=true}]
60

In the following example, when traversing the Entity containing components A and B, and delete the B component of the Entity in other thread.

EntityRepository entityRepository = (EntityRepository) new EntityRepository.Factory().create("test");
         final var list = new ArrayList<Entity>(100);
         for (int i = 0; i < 100; i++) {
             list.add(entityRepository.createEntity(new A(i), new B(i)));
         }
         Scheduler scheduler = entityRepository.createScheduler();

         scheduler.schedule(() -> entityRepository.findEntitiesWith(A.class, B.class).forEach(rs -> {
             try {
                 System.out.println(rs);
                 TimeUnit.MILLISECONDS.sleep(10);
             } catch (InterruptedException e) {
                 throw new RuntimeException(e);
             }
         }));


         new Thread(() -> {
             try {
                 TimeUnit.MILLISECONDS.sleep(10);
                 for (int i = 0; i < 100; i++) {
                     list.get(i).removeType(B.class);
                 }
             } catch (InterruptedException e) {
                 throw new RuntimeException(e);
             }
         }).start();
        
         scheduler.tick();

The result is

With2[comp1=A[id=99], comp2=B[id=99], entity=Entity={id=|0:2:99|-> [A[id=99]], stateId=|1:0:0|, enabled=true}]
With2[comp1=A[id=98], comp2=B[id=98], entity=null]
With2[comp1=A[id=97], comp2=B[id=97], entity=null]
With2[comp1=A[id=96], comp2=B[id=96], entity=null]
With2[comp1=A[id=95], comp2=B[id=95], entity=null]
With2[comp1=A[id=94], comp2=B[id=94], entity=null]
With2[comp1=A[id=93], comp2=B[id=93], entity=null]
With2[comp1=A[id=92], comp2=B[id=92], entity=null]
With2[comp1=A[id=91], comp2=B[id=91], entity=null]
With2[comp1=A[id=90], comp2=B[id=90], entity=null]
With2[comp1=A[id=89], comp2=B[id=89], entity=null]
With2[comp1=A[id=88], comp2=B[id=88], entity=null]
With2[comp1=A[id=87], comp2=B[id=87], entity=null]
With2[comp1=A[id=86], comp2=B[id=86], entity=null]
With2[comp1=A[id=85], comp2=B[id=85], entity=null]
With2[comp1=A[id=84], comp2=B[id=84], entity=null]
With2[comp1=A[id=83], comp2=B[id=83], entity=null]
With2[comp1=A[id=82], comp2=B[id=82], entity=null]
With2[comp1=A[id=81], comp2=B[id=81], entity=null]
With2[comp1=A[id=80], comp2=B[id=80], entity=null]
With2[comp1=A[id=79], comp2=B[id=79], entity=null]
With2[comp1=A[id=78], comp2=B[id=78], entity=null]
With2[comp1=A[id=77], comp2=B[id=77], entity=null]
With2[comp1=A[id=76], comp2=B[id=76], entity=null]
With2[comp1=A[id=75], comp2=B[id=75], entity=null]
With2[comp1=A[id=74], comp2=B[id=74], entity=null]
With2[comp1=A[id=73], comp2=B[id=73], entity=null]
With2[comp1=A[id=72], comp2=B[id=72], entity=null]
With2[comp1=A[id=71], comp2=B[id=71], entity=null]
With2[comp1=A[id=70], comp2=B[id=70], entity=null]
With2[comp1=A[id=69], comp2=B[id=69], entity=null]
With2[comp1=A[id=68], comp2=B[id=68], entity=null]
With2[comp1=A[id=67], comp2=B[id=67], entity=null]
With2[comp1=A[id=66], comp2=B[id=66], entity=null]
With2[comp1=A[id=65], comp2=B[id=65], entity=null]
With2[comp1=A[id=64], comp2=B[id=64], entity=null]
With2[comp1=A[id=63], comp2=B[id=63], entity=null]
With2[comp1=A[id=62], comp2=B[id=62], entity=null]
With2[comp1=A[id=61], comp2=B[id=61], entity=null]
With2[comp1=A[id=60], comp2=B[id=60], entity=null]
With2[comp1=A[id=59], comp2=B[id=59], entity=null]
With2[comp1=A[id=58], comp2=B[id=58], entity=null]
With2[comp1=A[id=57], comp2=B[id=57], entity=null]
With2[comp1=A[id=56], comp2=B[id=56], entity=null]
With2[comp1=A[id=55], comp2=B[id=55], entity=null]
With2[comp1=A[id=54], comp2=B[id=54], entity=null]
With2[comp1=A[id=53], comp2=B[id=53], entity=null]
With2[comp1=A[id=52], comp2=B[id=52], entity=null]
With2[comp1=A[id=51], comp2=B[id=51], entity=null]
With2[comp1=A[id=50], comp2=B[id=50], entity=null]
With2[comp1=A[id=50], comp2=B[id=50], entity=null]
With2[comp1=A[id=51], comp2=B[id=51], entity=null]
With2[comp1=A[id=52], comp2=B[id=52], entity=null]
With2[comp1=A[id=53], comp2=B[id=53], entity=null]
With2[comp1=A[id=54], comp2=B[id=54], entity=null]
With2[comp1=A[id=55], comp2=B[id=55], entity=null]
With2[comp1=A[id=56], comp2=B[id=56], entity=null]
With2[comp1=A[id=57], comp2=B[id=57], entity=null]
With2[comp1=A[id=58], comp2=B[id=58], entity=null]
With2[comp1=A[id=59], comp2=B[id=59], entity=null]
With2[comp1=A[id=60], comp2=B[id=60], entity=null]
With2[comp1=A[id=61], comp2=B[id=61], entity=null]
With2[comp1=A[id=62], comp2=B[id=62], entity=null]
With2[comp1=A[id=63], comp2=B[id=63], entity=null]
With2[comp1=A[id=64], comp2=B[id=64], entity=null]
With2[comp1=A[id=65], comp2=B[id=65], entity=null]
With2[comp1=A[id=66], comp2=B[id=66], entity=null]
With2[comp1=A[id=67], comp2=B[id=67], entity=null]
With2[comp1=A[id=68], comp2=B[id=68], entity=null]
With2[comp1=A[id=69], comp2=B[id=69], entity=null]
With2[comp1=A[id=70], comp2=B[id=70], entity=null]
With2[comp1=A[id=71], comp2=B[id=71], entity=null]
With2[comp1=A[id=72], comp2=B[id=72], entity=null]
With2[comp1=A[id=73], comp2=B[id=73], entity=null]
With2[comp1=A[id=74], comp2=B[id=74], entity=null]
With2[comp1=A[id=75], comp2=B[id=75], entity=null]
With2[comp1=A[id=76], comp2=B[id=76], entity=null]
With2[comp1=A[id=77], comp2=B[id=77], entity=null]
With2[comp1=A[id=78], comp2=B[id=78], entity=null]
With2[comp1=A[id=79], comp2=B[id=79], entity=null]
With2[comp1=A[id=80], comp2=B[id=80], entity=null]
With2[comp1=A[id=81], comp2=B[id=81], entity=null]
With2[comp1=A[id=82], comp2=B[id=82], entity=null]
With2[comp1=A[id=83], comp2=B[id=83], entity=null]
With2[comp1=A[id=84], comp2=B[id=84], entity=null]
With2[comp1=A[id=85], comp2=B[id=85], entity=null]
With2[comp1=A[id=86], comp2=B[id=86], entity=null]
With2[comp1=A[id=87], comp2=B[id=87], entity=null]
With2[comp1=A[id=88], comp2=B[id=88], entity=null]
With2[comp1=A[id=89], comp2=B[id=89], entity=null]
With2[comp1=A[id=90], comp2=B[id=90], entity=null]
With2[comp1=A[id=91], comp2=B[id=91], entity=null]
With2[comp1=A[id=92], comp2=B[id=92], entity=null]
With2[comp1=A[id=93], comp2=B[id=93], entity=null]
With2[comp1=A[id=94], comp2=B[id=94], entity=null]
With2[comp1=A[id=95], comp2=B[id=95], entity=null]
With2[comp1=A[id=96], comp2=B[id=96], entity=null]
With2[comp1=A[id=97], comp2=B[id=97], entity=null]
With2[comp1=A[id=98], comp2=B[id=98], entity=null]
With2[comp1=A[id=99], comp2=B[id=99], entity=null]

@endison1986
Copy link
Contributor Author

is this a bug? @enricostara

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@endison1986 @enricostara