-
Notifications
You must be signed in to change notification settings - Fork 99
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fail to generate perfect hash for reading when there are one or more unicode keys in struct #666
Comments
Thanks for this bug report. Could you share some of the unicode keys that are failing for you? I haven't tested much with unicode keys, so I'll have to look into it and add more unit tests. |
Sure, the struct for one of my csv files struct FishRecord
{
float Duration;
float FishSize;
std::uint8_t Amount;
std::string FishBaitName;
std::string SurfaceSlapFishName;
std::string MoochFishName;
std::string BuffName;
std::string FishingSpotPlaceName;
std::string BiteTypeName;
std::string CaughtFishName;
std::string HooksetName;
};
template <>
struct glz::meta<FishRecord>
{
using T = FishRecord;
static constexpr auto value = object("上钩的鱼", &T::CaughtFishName, //
"间隔", &T::Duration, //
"尺寸", &T::FishSize, //
"数量", &T::Amount, //
"鱼饵", &T::FishBaitName, //
"拍水的鱼", &T::SurfaceSlapFishName, //
"以小钓大的鱼", &T::MoochFishName, //
"Buff", &T::BuffName, //
"钓场", &T::FishingSpotPlaceName, //
"咬钩类型", &T::BiteTypeName, //
"提钩类型", &T::HooksetName //
);
}; |
I went ahead and tested a little bit, "when there are one or more unicode keys in struct" doesn't seem to be the case. template <>
struct glz::meta<unicode_keys_t>
{
using T = unicode_keys_t;
static constexpr auto value = object("简体汉字", &T::field0, // simplified chinese characters
"漢字寿限無寿限無五劫", &T::field1, // traditional chinese characters / kanji
"こんにちはむところやぶらこうじのぶらこうじパイポパイポパイポのシューリンガンシューリンガンのグーリンダイグーリンダイのポンポコピーのポンポコナの", &T::field2, // katakana
"한국인", &T::field3, // korean
"русский", &T::field4, // cyrillic
"สวัสดี", &T::field5, // thai
"english", &T::field6
);
}; however, when I replace field2 with |
@Nukoooo, I'm looking into this now. I'm wondering how you're using these structures with CSV reading. Glaze is designed to read into containers like If you can clarify your use case a little bit more I think we can resolve this. Thanks! |
Ah, I just realized I was providing the struct I used for testing and I'm sorry for that.
No, it was for testing in JSON, because I originally thought it was because of the CSV format so I tried removing I was reading into struct FishRecord
{
std::vector<float> Duration;
std::vector<float> FishSize;
std::vector<std::uint8_t> Amount;
std::vector<std::string> FishBaitName;
std::vector<std::string> SurfaceSlapFishName;
std::vector<std::string> MoochFishName;
std::vector<std::string> BuffName;
std::vector<std::string> FishingSpotPlaceName;
std::vector<std::string> BiteTypeName;
std::vector<std::string> CaughtFishName;
std::vector<std::string> HooksetName;
std::vector<std::string> IsLargeSizeName;
std::vector<std::string> IsCollectableName;
};
template <>
struct glz::meta<FishRecord>
{
using T = FishRecord;
static constexpr auto value = object("上钩的鱼", &T::CaughtFishName, //
"间隔", &T::Duration, //
"尺寸", &T::FishSize, //
"数量", &T::Amount, //
"鱼饵", &T::FishBaitName, //
"拍水的鱼", &T::SurfaceSlapFishName, //
"以小钓大的鱼", &T::MoochFishName, //
"Buff", &T::BuffName, //
"钓场", &T::FishingSpotPlaceName, //
"咬钩类型", &T::BiteTypeName, //
"提钩类型", &T::HooksetName, //
"大尺寸", &T::IsLargeSizeName, //
"收藏品", &T::IsCollectableName //
);
};
struct FishRecordRaw
{
std::vector<float> Duration;
std::vector<float> FishSize;
std::vector<std::uint8_t> Amount;
std::vector<std::int32_t> FishBaitIds;
std::vector<std::int32_t> SurfaceSlapFishIds;
std::vector<std::int32_t> MoochFishIds;
std::vector<std::int32_t> BuffFlags;
std::vector<std::int32_t> FishingSpotPlaceNameId;
std::vector<std::uint8_t> BiteType;
std::vector<std::uint32_t> CaughtFishId;
std::vector<std::uint8_t> Hookset;
std::vector<bool> IsLargeSize;
std::vector<bool> IsCollectable;
}
template <>
struct glz::meta<FishRecordRaw>
{
using T = FishRecordRaw;
static constexpr auto value = object(
"上钩的鱼", &T::CaughtFishId, //
"间隔", &T::Duration, //
"尺寸", &T::FishSize, //
"数量", &T::Amount, //
"鱼饵", &T::FishBaitIds, //
"拍水的鱼", &T::SurfaceSlapFishIds, //
"以小钓大的鱼", &T::MoochFishIds, //
"Buff", &T::BuffFlags, //
"钓场", &T::FishingSpotPlaceNameId, //
"咬钩类型", &T::BiteType, //
"提钩类型", &T::Hookset, //
"大尺寸", &T::IsLargeSize, //
"收藏品", &T::IsCollectable //
);
}; The user readable csv data:
The raw data:
|
Thanks for this. I made some progress today figuring it out. It has to do with the |
@Nukoooo, I think I fixed the bug with #670. The issue was with hashing unicode characters, I was casting If you run into more issues, please let me know. And, thanks so much for pointing this out! |
I was trying to read csv files which have unicode in keys and got this bug and debugged a little bit, turned out it was because of the unicode keys.
Using compiler: MSVC 19.38.33133 / Visual studio 2022 17.8.3 (it also doesn't work in 17.8.2)
Callstack from visual studio error report:
Code to reproduce
The text was updated successfully, but these errors were encountered: