Question: Can I infer the size of the component type when visiting storage pools?

[Ramito]

Hello!

I am trying to generically (without upfront knowledge of the component types) read the byte contents of all existing pools.

Is this possible?

I assume the only way I can access the pools generically is by using visit:

auto visitor = [](entt::sparse_set& pool) {
    // Can I get the pool element size?
};
entity_registry_.visit(visitor);

(BTW, the documentation has this example but it seems to me this is no longer possible:

// create a copy of an entity component by component
for(auto &&curr: registry.storage()) {
    if(auto &storage = curr.second; storage.contains(src)) {
        storage.emplace(dst, storage.get(src));
    }
}

)

[skypjack]

the documentation has this example but it seems to me this is no longer possible

Quite the opposite, this is possible since v3.9, that is the last version available upstream.
As for your first question instead, in the generic pool you've a get function that returns a void * at the moment. You can use it and the type info object with a reflection system or similar. This should address your needs.
Let me know if you've still doubts.

[Ramito]

@skypjack Thanks for the reply!

Hmmm... I am actually using v3.9!

But if I make a basic registry I cannot call storage without a template parameter. What may I be missing?

Back to my question, I am very sorry but I think I do need more details. I assume that the type info would be what I get by pool.type() (if pool is as my example above), but I have no idea how to get the size info from there!

[skypjack]

What may I be missing?

🤔
I don't know honestly, this is the code I've used for a test in v3.9 and I'm pretty sure it works since the CI isn't complaining. 😅
Maybe a CMake project or similar where you've updated the version but that doesn't refresh the local copy after the boostrap? Really dunno, just thinking aloud, it's hard to say from here.

As for your question, the type info can help you to find a meta type with which you can elaborate the void *.
Long story short, pools are lazily instantiated in EnTT and their types are unknown (until the user says - see, this is a T). Therefore, something like a reflection system acts as glue code when you want to work with opaque data.
Can I ask you what's your goal? I mean, why do you need the size? Maybe this is an YZ-problem and I can help more if you tell me what you're trying to achieve. 🙂

[Ramito]

If registry.storage() is supposed to compile with 3.9 seems I need to dig around. This is a fresh copy of entt for this project, not an upgrade, but I wonder if I did not setup CMake correctly. Unless I need to include a particular header other than entt.hpp. Anyway, at least now I know!

What I am trying to do is to iterate all the component pools and read the byte data directly without manually registering this code per component type. I am still experimenting but I have two concrete goals right now: Compute a checksum for all existing components and copy entities from one registry to the other without knowing upfront which components where registered.

I can elaborate on why I want to do the later, but I think that goes beyond the specific question.

I am not very familiar with reflection on C++, is there any resource you'd recommend to get me started?

Thank you!

[skypjack]

Unless I need to include a particular header other than entt.hp

Uhm... is it the single include file? Maybe it's out-of-date? Otherwise, the entt.hpp file under entt is just a list of includes and should contain the right version of the registry.
I'm cursious now btw, let me know if you find what the problem is.

What I am trying to do is ...

To copy the storage, at the moment you can iterate the pool, then get and emplace entities and values one at a time.
I'm working to make a sort of get-and-emplace-everything but it's not in v3.9, I'm sorry. However, you can have it with some reflection.
C++ doesn't support reflection btw. It's planned for C++23 but it's not even confirmed at the moment.
EnTT offers a runtime reflection module under the meta directory. So, for example, you could:

• Create a meta type for T and register a couple of functions to compute the checksum and blindly copy everything across registries
• Run ... run ... run
• Iterate the pools on the source registry, get the type info and use it to retrieve the meta type, then pass the pool or the value to the right meta functions and get the job done

If this sounds like something that could work for you, I can prepare a super small example if you like. Let me know. 👋

[Ramito]

This is certainly of interest to me, so I would very much appreciate an example!
Thank you very much for your help! (And the library of course!)

[skypjack]

Sorry, I've been busy today and I haven't found the time to pack a proper example.
Here you can find a snippet that shows what I mean with copying a storage using meta.
Let me know if it's clear enough or if you have any doubt. I'll try to keep up as soon as possible. 👍

[Ramito]

This does let me move forward, thank you very much for the example!

Purely for academic interest, what I was hoping for is a way to do this without having to upfront register my components for this, as you do in your example (register_meta_type_for<position>();). I wanted to be able to copy all components that exist in one registry without having forward knowledge of their types. I assumed that the pools would need to know the object size implicitly.

I have not looked into my issues running the untyped storage, but if I find it I will let you know!

Thanks!

[skypjack]

Well, technically speaking it's possible already as:

for(auto [id, storage]: src_registry.storage()) {
    auto &&other = dst_registry.storage(storage.type());

    for(auto entity: storage) {
        other.emplace(entity, storage.get(entity);
    }
}

Morever, as I said before, I'm working on something to make it a single line instruction. However, this is only meant to work across storage classes and not across registries.
The fact is that EnTT is designed in such a way that you don't need to use the registry. You can just use the storage classes and that's it. So, the latter has not the knowledge about the registry, for obvious reasons. Therefore, the biggest problem to address is how to create the storage in an empty registry from an opaque reference to a base class (that is, when you don't have the type).
The next version should fill this gap. The solution will likely be to inject a storage class that you obtain as a copy from an existing pool. However, this won't work across registries with different entity types.

[Ramito]

Hey!

So that snippet looks exactly like what I need at this moment (still experimenting!). While looking at it, though, I had to figure out why I cannot access storage with no template arguments. The reasons seems to be that the version I got is the one from here: https://github.com/skypjack/entt/releases/tag/v3.9.0

The commit that adds that method was done the next day! :P

[skypjack]

🤔

Wait a moment. This is from v3.9:

[[nodiscard]] auto storage() ENTT_NOEXCEPT {
    return iterable_adaptor{internal::storage_proxy_iterator{pools.begin()}, internal::storage_proxy_iterator{pools.end()}};
}

That is, this is possible with v3.9 already:

for(auto [id, storage]: src_registry.storage()) {
    // ...
}

I really don't know why you don't see it but, well, it's there. 😅

[Ramito]

You might want to be aware that this link:

leads to a version without it.

Thanks for all the responses so far!

[skypjack]

😲 Oh, this I don't know actually, I've always referenced the git tag. What's wrong there? The single include file or the tarball?
I remember it happened one more time too. If I'm not mistaken, it's wrong when I create the release before the git tag or something like that. 🤔

EDIT

The commit points to this version of the registry and the storage method is there. The tarball contains the same method too.
I think I need more details about what's wrong with that link, I'm sorry. 😞

[Ramito]

Hello!

A few updates on this. Finally got around field testing these ideas. First is that your example above needs some tweaks:

Your example:

for(auto [id, storage]: src_registry.storage()) {
    auto &&other = dst_registry.storage(storage.type());

    for(auto entity: storage) {
        other.emplace(entity, storage.get(entity);
    }
}

My (compiling) code:

for (auto pair : source.storage()) {
	const auto& source_store = pair.second;
	auto& target_store = *target.storage(source_store.type().hash());
	for (auto entity : source_store) {
		target_store.emplace(entity, source_store.get(entity));
	}
}

I had to use source_store.type().hash(), because source_store.type() is not the right type, and the id is otherwise private. This works, but using hash feels like a hack, as there really should be no guarantee this will work the same in the future.

Another issue I discovered is that if the target registry has not had the component type initialized this will result in null memory access. I think I can deal with this, though.

[chengts95]

auto &&other = dst_registry.storage(storage.type()); only works when dst_registry already has the storage. Otherwise, it returns zero. i was expecting dst_registry.storage(storage.type()) can create such storage in the dst_registry.

[Ramito]

Thank you @chengts95!

Any recommendations on how to make sure the storage has the pool, sort of adding/removing a component just to be sure?

I would like to use assure for example, but it is a private member.

[chengts95]

To put storage in the pool, you have to use a template because C++ types don't exist at runtime.

[Ramito]

Thank you for the reply.

I understand that. My question is what would be a good way to create the pool on the target registry (yes using type specific templates) if no entity on the target registry has received the component yet.

Currently, to get around this I am adding and removing a component of the type on a fake entity, but I wonder if there is a call for such purpose, along the lines of registry.touch<TComponent>().

[Ramito]

This is what I am settling with for now:

auto& target_storage = target_registry.storage<TComponent>();
// Hack: Ensure pool exists in target registry
if (target_storage.capacity() == 0) {
    target_storage.clear();
}

The rest (the transfer part) is as above, and as a proof of concept this works pretty well.

[skypjack]

Yeah, the non-template storage method is only available upstream and isn't part of any release yet. The goal is to add all is needed to help creating pools across registries. However, it won't be a method in the storage class, mainly because the latter doesn't know about the registry and this thing isn't going to change any time soon.
Probably, the final solution will be a fake vtable directly provided by the registry, something along this line:

for (auto [id, source]: registry.storage()) {
	auto& target = registry.vtable_for(source.type().hash()).emplace(other_registry);
        // ...
}

Though, take this with a grain of salt at the moment, I really don't know how it will look like (and any suggestions is welcome in this sense 🙂).

[Ramito]

Thank you, @skypjack.

Looking forward to that!

Any recommended alternatives to my hack above in the mean time?

[skypjack]

@Ramito registry.storage<T>() is enough to create it.

[Ramito]

@skypjack but then if I don't do anything with it I get a no-discard error, since it gets compiled out. I think it'd be nice to have a call to just ensure it's there without any intention to sue it at the time.

[skypjack]

Yeah, I want to remove the [[nodiscard]] in this case. I'm using it to create pools too and it's getting annoying. 😅

[Ramito]

😁 Thanks!

[chengts95]

this thing isn't going to change any time so

Maybe that is true, a vtable for constructing storage by type id. I believe the main problem is to new a proper sparse_set with type id only and automate this process. It is true that we can put the class constructor into the registry's storage vtable when new the storage. But the instance in vtable must have access to other registry's private members to emplace the storage.
Maybe add something like this to the registry?

   
    void* registry::emplace_storage(const id_type id , function_ptr  storage_constructor) {

        auto &&cpool = pools[id];

        if(!cpool) {
            cpool.reset( storage_constructor());
            cpool->bind(forward_as_any(*this));
        }

        return  cpool.get();
    }

[chengts95]

This is how I implement the storage construction

void copyReg(const entt::registry &registry, entt::registry &other)
    {
        other.assign(registry.data(), registry.data() + registry.size(), registry.released());
        for (const auto [id, storage] : registry.storage())
        {
            auto newstorage = registry.construct_storage(id);

            if (newstorage)
            {
                auto &&dst = other.emplace_or_replace_storage(id, newstorage);

                if (dst)
                {
                    for (auto &&i : storage)
                    {
                        dst->emplace(i, storage.get(i)); //i have to use this way because insert missed some entities.
                    }
                }
            }
        }
    }

//registry.hpp
//...
//private:
//    dense_hash_map<id_type, base_type *(*)(), identity> vtable;
    base_type *emplace_or_replace_storage(const id_type id, base_type *storage) {
        auto &&cpool = pools[id];

        // if(!cpool) {
        cpool.reset(storage);
        cpool->bind(forward_as_any(*this));
        //}

        return cpool.get();
    }
    base_type *construct_storage(const id_type id) const {
        const auto f = vtable.find(id);
        if(f != vtable.end()) {
            auto func = f->second;
            return func();
        }

        return nullptr;
    }

    template<typename Component>
    [[nodiscard]] auto &assure(const id_type id = type_hash<Component>::value()) {
        static_assert(std::is_same_v<Component, std::decay_t<Component>>, "Non-decayed types not allowed");
        auto &&cpool = pools[id];

        if(!cpool) {
            vtable[id] = []() { return static_cast<base_type *>(new storage_type<Component>{}); };
            cpool.reset(new storage_type<Component>{});
            cpool->bind(forward_as_any(*this));
        }

        return static_cast<storage_type<Component> &>(*cpool);
    }

The problem is the vtable need to be reproduced in other registry too...

[skypjack]

Mmm probably the vtable should work the other way around, that is:

registry.vtable_for(id).setup(other_registry);

This way you have a function with full knowledge that triggers a .storage<T> call on other_registry.
Actually, the function could even be as simple as &storage<T> (if it only returned an opaque storage 😅).

[chengts95]

Mmm probably the vtable should work the other way around, that is:

registry.vtable_for(id).setup(other_registry);

This way you have a function with full knowledge that triggers a .storage<T> call on other_registry. Actually, the function could even be as simple as &storage<T> (if it only returned an opaque storage 😅).

I remember the constructor's pointer cannot be obtained, so I save a lambda function to hashmap. I don't know what is the return type or the instance's location of that vtable_for(). For me, i only did this other_registry.set_vtable(registry.get_vtable()). I only encountered a strange behavior of storage.insert, it skipped some components while storage.emplace has no problem.

I have tested my copyReg function in my application and it works well.

[khalv]

What's the current status on duplicating an entity with all components (possibly to a different registry)? None of the examples listed even compile for me with 3.9.0.

This one:

for (auto pair : source.storage()) {
	const auto& source_store = pair.second;
	auto& target_store = dest.storage(source_store.type().hash());
	for (auto entity : source_store) {
		target_store.emplace(entity, source_store.get(entity));
	}
}

Fails on this line, specifically the argument of the storage method:
auto& target_store = dest.storage(source_store.type().hash());

And it's the same on this version:

for(auto [id, storage]: src_registry.storage()) {
    auto &&other = dst_registry.storage(storage.type());

    for(auto entity: storage) {
        other.emplace(entity, storage.get(entity));
    }
}

Also, from what i understand from the discussion, this method only works if the destination storage has already been initialized by creating a component of that type. Is there any alternative solution?

[skypjack]

You're right, this is available upstream but not part of v3.9:

auto& target_store = dest.storage(source_store.type().hash());

While this:

this method only works if the destination storage has already been initialized by creating a component of that type

Hopefully will be addressed before the next version. With v3.9 you can get close using the storage type function to get a type info object to use to retrieve a meta type, if any.

[danielytics]

I've been using poly_storage to copy entities between registries (both to copy entire registries, which I do from the editors registry to the games registry when you press "play", and also for single entities, which I use for "prototype"/prefab entities that can be instantiated by copying from the prototype to the live entities). I do it like this and it seems to work well enough, for the past ~10 months.

I'm wondering how these new changes impact that? Will I be able to implement this then without custom poly_storage? Any tradeoffs I should know about?

Thanks :)

[skypjack]

It turned out that this solution is waaaaaay simpler but also incomplete in a sense. Something that I'll fix soon but still.
Briefly, if you use meta, it's almost a direct mapping between the poly storage and the meta type. Otherwise, the opaque get and emplace functions make you copy components across registries blindly and easily.
The only issue if compared to the poly storage is that the blind copy doesn't create a storage class in the target registry for you. The fix that I've mentioned above aims to fill the gap.

[skypjack]

Funny enough, after struggling with this feature for a while I realized that EnTT already offers everything needed to get what I want in a flexible and customizable way and without library additions.
I'm updating the entity_copy.cpp example to get rid of the infamous TODO 🙂 then I'll slightly review the entity storage machinery to make it cleaner because it requires some non-obvious knowledge at the moment.

CC @Qix- who put the TODO in the entity_copy.cpp file. 👍