✨ Add PETSCII, KOI8-R, GBK, GB18030, Big5-HKSCS, a slew of Windows Co…

…de Pages
soasis · May 16, 2023 · 0b78eda · 0b78eda
1 parent 6c5d7ba
commit 0b78eda
Show file tree

Hide file tree

Showing 70 changed files with 3,930 additions and 531 deletions.
diff --git a/CMakeLists.txt b/CMakeLists.txt
@@ -225,6 +225,7 @@ if (ZTD_TEXT_EXAMPLES OR ZTD_TEXT_BENCHMARKS)
 		# simdutf
 		set(SIMDUTF_BENCHMARKS OFF)
 		set(SIMDUTF_SINGLE_HEADER OFF)
+		set(SIMDUTF_TESTS OFF)
 		set(SIMDUTF_TOOLS OFF)
 		set(BUILD_TESTING OFF)
 		FetchContent_Declare(

diff --git a/...api/encodings/ibm_424_hebrew_bulletin.rst → ...ation/source/api/encodings/big5_hkscs.rst b/...api/encodings/ibm_424_hebrew_bulletin.rst → ...ation/source/api/encodings/big5_hkscs.rst
@@ -28,17 +28,17 @@
 ..
 .. =============================================================================>
 
-IBM Codepage 424 - Hebrew (Bulletin)
-====================================
+Big5 Hong Kong Supplementary Character Set (HKSCS)
+==================================================
 
 
 
-.. doxygenvariable:: ztd::text::ibm_424_hebrew_bulletin
+.. doxygenvariable:: ztd::text::big5_hkscs
 
 
 
-Base Template
--------------
+Base Templates
+--------------
 
-.. doxygenclass:: ztd::text::basic_ibm_424_hebrew_bulletin
+.. doxygenclass:: ztd::text::basic_big5_hkscs
 	:members:
diff --git a/...on/source/api/encodings/ibm_1006_urdu.rst → ...ation/source/api/encodings/euc_kr_uhc.rst b/...on/source/api/encodings/ibm_1006_urdu.rst → ...ation/source/api/encodings/euc_kr_uhc.rst
@@ -28,17 +28,18 @@
 ..
 .. =============================================================================>
 
-IBM Codepage 1006 - Urdu
-========================
+EUC-KR (Unified Hangul Code)
+============================
 
 
 
-.. doxygenvariable:: ztd::text::ibm_1006_urdu
 
+.. doxygenvariable:: ztd::text::euc_kr_uch
 
 
-Base Template
--------------
 
-.. doxygenclass:: ztd::text::basic_ibm_1006_urdu
+Base Templates
+--------------
+
+.. doxygenclass:: ztd::text::basic_euc_kr_uch
 	:members:
diff --git a/...n/source/api/encodings/ibm_856_hebrew.rst → ...entation/source/api/encodings/gb18030.rst b/...n/source/api/encodings/ibm_856_hebrew.rst → ...entation/source/api/encodings/gb18030.rst
@@ -28,17 +28,18 @@
 ..
 .. =============================================================================>
 
-IBM Codepage 856 - Hebrew
-=========================
+GB18030
+=======
 
+An encoding capable of handling all known Unicode-encoded characters, and occasionally a few more (the most recent version of Unicode covers all values available in the most up-to-date GB-18030).
 
 
-.. doxygenvariable:: ztd::text::ibm_856_hebrew
+.. doxygenvariable:: ztd::text::gb18030
 
 
 
-Base Template
--------------
+Base Templates
+--------------
 
-.. doxygenclass:: ztd::text::basic_ibm_856_hebrew
+.. doxygenclass:: ztd::text::basic_gb18030
 	:members:
diff --git a/...ntation/source/api/encodings/atari_st.rst → documentation/source/api/encodings/gbk.rst b/...ntation/source/api/encodings/atari_st.rst → documentation/source/api/encodings/gbk.rst
@@ -28,17 +28,18 @@
 ..
 .. =============================================================================>
 
-Atari ST
-========
+GBK
+===
 
+A legacy encoding typically for Chinese languages.
 
 
-.. doxygenvariable:: ztd::text::atari_st
+.. doxygenvariable:: ztd::text::gbk
 
 
 
-Base Template
--------------
+Base Templates
+--------------
 
-.. doxygenclass:: ztd::text::basic_atari_st
+.. doxygenclass:: ztd::text::basic_gbk
 	:members:
diff --git a/documentation/source/api/encodings/ibm_866_cyrillic.rst b/documentation/source/api/encodings/ibm_866_cyrillic.rst
diff --git a/documentation/source/api/encodings/kazakh_strk1048.rst b/documentation/source/api/encodings/kazakh_strk1048.rst
diff --git a/documentation/source/api/encodings/koi8_u.rst b/documentation/source/api/encodings/koi8_u.rst
diff --git a/documentation/source/api/encodings/petscii.rst b/documentation/source/api/encodings/petscii.rst
@@ -0,0 +1,52 @@
+.. =============================================================================
+..
+.. ztd.text
+.. Copyright © 2022-2023 JeanHeyd "ThePhD" Meneide and Shepherd's Oasis, LLC
+.. Contact: opensource@soasis.org
+..
+.. Commercial License Usage
+.. Licensees holding valid commercial ztd.text licenses may use this file in
+.. accordance with the commercial license agreement provided with the
+.. Software or, alternatively, in accordance with the terms contained in
+.. a written agreement between you and Shepherd's Oasis, LLC.
+.. For licensing terms and conditions see your agreement. For
+.. further information contact opensource@soasis.org.
+..
+.. Apache License Version 2 Usage
+.. Alternatively, this file may be used under the terms of Apache License
+.. Version 2.0 (the "License") for non-commercial use; you may not use this
+.. file except in compliance with the License. You may obtain a copy of the
+.. License at
+..
+.. https://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+..
+.. =============================================================================>
+
+PETSCII (Shifted & Unhsifted, Combined) / CBM ASCII
+===================================================
+
+PET Standard Code of Information Interchange (PETSCII) was used for Commodore Buisiness Maschines and then moved into other Commodore machines (and adjacent machines). It has a "shifted" version (when the shift key was held) and an "unshifted" version (when the shift key was not being held).
+
+The state object for this encoding contains an enumeration that allows the user to select the shifted or unshifted versions at-will.
+
+
+
+.. doxygenclass:: ztd::text::petscii_state
+
+.. doxygenenum:: ztd::text::petscii_shift
+
+.. doxygenvariable:: ztd::text::petscii
+
+
+
+Base Templates
+--------------
+
+.. doxygenclass:: ztd::text::basic_petscii
+	:members:
diff --git a/documentation/source/api/encodings/shift_jisx0208.rst b/documentation/source/api/encodings/shift_jisx0208.rst
@@ -0,0 +1,50 @@
+.. =============================================================================
+..
+.. ztd.text
+.. Copyright © 2022-2023 JeanHeyd "ThePhD" Meneide and Shepherd's Oasis, LLC
+.. Contact: opensource@soasis.org
+..
+.. Commercial License Usage
+.. Licensees holding valid commercial ztd.text licenses may use this file in
+.. accordance with the commercial license agreement provided with the
+.. Software or, alternatively, in accordance with the terms contained in
+.. a written agreement between you and Shepherd's Oasis, LLC.
+.. For licensing terms and conditions see your agreement. For
+.. further information contact opensource@soasis.org.
+..
+.. Apache License Version 2 Usage
+.. Alternatively, this file may be used under the terms of Apache License
+.. Version 2.0 (the "License") for non-commercial use; you may not use this
+.. file except in compliance with the License. You may obtain a copy of the
+.. License at
+..
+.. https://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+..
+.. =============================================================================>
+
+SHIFT-JISX0208
+==============
+
+The version of SHIFT-JIS that corresponds to standard X0208, one of the more prevalent versions.
+
+Note that many versions of SHIFT-JIS do not correspond to one standard and often have different interpretations or characteristics. The communities which use them label them, indiscriminatly, as SHIFT-JIS without any kind of specific indicator or even out-of-band modifier. The text community surrounding this is, with all due respect, one gigantic mess. Most industry professionals inside and outside of Japan dealing with such text tend to gravitate towards the SHIFT-JISX0208 release, and simply use replacement characters / invalid indicators for such input text.
+
+As such, it is advisable to perhaps attempt to find some out-of-band data to see if a specific data is, indeed, meant to be SHIFT-JISX0208.
+
+
+
+.. doxygenvariable:: ztd::text::shift_jisx0208
+
+
+
+Base Templates
+--------------
+
+.. doxygenclass:: ztd::text::basic_shift_jisx0208
+	:members: