Skip to content

[C++] To support the utf8 utf16 and using simd to accelerate the optimization #1964

@pandalee99

Description

@pandalee99

Feature Request

Earlier, I implemented utf16 to utf8 and used simd to speed up the process.

std::string utf16ToUtf8(const std::u16string &utf16, bool is_little_endian)

Now, you also need to implement utf8 to utf16.
Because now in java, using utf8, 16, latin1 is becoming very common, now we need to implement it in py as well.
But for speed reasons, we might consider using a direct call to a c++ module.

For simd acceleration, it is necessary to adapt to x86 architecture, arm and risc-v.
The solution is that you can use the avx2 instruction set under x86 and so on...

Is your feature request related to a problem? Please describe

Module lack utf8 utf16.

Describe the solution you'd like

Implement utf8 to utf16, and use simd acceleration.

Describe alternatives you've considered

No response

Additional context

No response

Metadata

Metadata

Assignees

Labels

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions