PersistenceException: TypeId conflict for 1 byte on AWS S3 (and nowhere else)

[dbucek]

Environment Details

• EclipseStore Version: 3.1.0
• JDK version: 21.0.9
• OS: Docker, eclipse-temurin-21-alpine
• Used frameworks: Spring Boot 4.0.0

Describe the bug

Application run inside on AWS EC2 and EclipseStore in AWS S3. When I run it for the first time it creates 95 PersistentTypeDictionary.[0-94] (see screenshot). So, far application works fine. BUT when I restart it (or stop and start again) it immediatelly throw PersistenceException: TypeId conflict for 1 byte.

But it's completely fine on local deployment. Even when I directly connect to the same AWS S3. It works good even I try to run Docker locally and connect to AWS S3. What is more strange if the first initialization is done by local or local docker everything works fine and it contains only one PersistentTypeDictionary.0 file.

aws-docker-exception.log

To Reproduce

Hard to say how to reproduce on different environment. On my environment it happens always.

Expected behavior

Just start up and works as usual.

Screenshots

Additional context

Nothing else on my mind yet :-(

[hg-ms]

The exception states that TypeId 1 already exists in the type-dictionary, a duplicate can’t be added.
This might be caused by more than one storage manager instance accessing the S3 concurrently which must not happen.

[dbucek]

I understand the first sentence. But I don't see any other storage manager instance running at the same time. Is there anything I can check? Debug logs?

What I see is normal startup, no difference in starting logs compare to non-AWS docker/local instance. I don't run multiple docker instances. Simply I have no clue why it is failing.

[hg-ms]

If no second storage instance is involved, it’s hard to say what’s wrong. I never saw that exception. Here is some additional info that may help identify the cause of the problem:

You can activate some logging using popular logging frameworks, eclipse store uses the slf4j api.
E.g. Using Logback you could enable loggers:

<logger name="org.eclipse.serializer" level = "debug"></logger>
<logger name="org.eclipse.store" level = "debug"></logger>

At the first startup ES write the type-dictionary to the storage target if not yet existing.
If the type-dictionary exists it will load it but not write it. Additional entries to the type-dictionary will be written only if new type is stored. The byte type usually is the first entry in the TD and sound never be added a second time. That’s why I assumed that there was a second active storage involved.

The Typedictionary is human readable text maybe you can check the content of the ptd files. They should sum up to the same content as a local PersistenceTypeDictionary.ptd.

If not yet done, you should also delete the S3 content after the error happens.