Replace text with unicode symbols #413

rowlesmr · 2023-06-04T01:51:07Z

This PR replace all unambiguous markup with unicode.

The eventual goal is to have unicode symbols everywhere, rather than textual descriptions.

I haven't touched the update dates yet.

Strictly, this should be tau, but I think it means theta.

ERROR: LoadError: StringIndexError: invalid index [21], valid nearby indices [20]=>'°', [22]=>''' Stacktrace: [1] string_index_err(s::Token, i::Int64) @ Base ./strings/string.jl:12 [2] SubString{Token}(s::Token, i::Int64, j::Int64) @ Base ./strings/substring.jl:32 [3] SubString @ ./strings/substring.jl:38 [inlined] [4] SubString @ ./strings/substring.jl:40 [inlined] [5] getindex @ ./strings/substring.jl:255 [inlined] [6] check_delimiter(value::Tree) @ Main ~/work/cif_core/cif_core/julia_cif_tools/layout.jl:514 [7] transformer_func(l::Linter, #unused#::Val{:scalar_item}, meta::Lerche.Meta, tree::Tree) @ Main ~/work/cif_core/cif_core/julia_cif_tools/layout.jl:491 [8] _call_userfunc(v::Linter, tree::Tree) @ Lerche ~/.julia/packages/Lerche/QxD5R/src/visitors.jl:269 [9] visit(v::Linter, tree::Tree) @ Lerche ~/.julia/packages/Lerche/QxD5R/src/visitors.jl:299 [10] visit(v::Linter, tree::Tree) (repeats 5 times) @ Lerche ~/.julia/packages/Lerche/QxD5R/src/visitors.jl:296 [11] lint_report(filename::String; ref_dic::String, import_dir::String, warn::Bool, no_text::Bool) @ Main ~/work/cif_core/cif_core/julia_cif_tools/linter.jl:50 [12] top-level scope @ ~/work/cif_core/cif_core/julia_cif_tools/linter.jl:107 in expression starting at /home/runner/work/cif_core/cif_core/julia_cif_tools/linter.jl:104 Error: Process completed with exit code 1.

This reverts commit 392cabc.

rowlesmr · 2023-06-04T03:10:48Z

See #414 re checker failure

jamesrhester · 2023-06-05T05:30:56Z

Unicode in the definition text is a little tricky, because this definition text is used to automatically prepare text for Volume G and the online web pages. Before doing this, we should coordinate with both Vol G people (me and @nautolycus ) and the Chester people (@publcif ) to confirm that they can handle the likely unicode characters that will pop up. So I'd hold off on doing anything until those people respond here. Last I checked the Vol G workflow would suffer.

rowlesmr · 2023-06-05T06:35:45Z

Roger dodger.

nautolycus · 2023-06-05T09:47:31Z

Last I checked the Vol G workflow would suffer.

This shouldn't be an issue for Vol. G, at least so long as Unicode is used conservatively (the Little Dictionary has sentences with Russian, Japanese and Eastern European text, which is a little tricky to handle!). I'm polling internal views to see if people here can identify any other possible gotchas, so don't commit just yet.

vaitkus

I am also not sure if values of _description_example.case should be modified since they provide examples on how values should be written in actual CIF instance files. While Unicode can be used in CIF2 files (and its use is probably encouraged), the given example could not be used verbatim in CIF1.1 files.

templ_attr.cif

vaitkus · 2023-06-14T12:16:58Z

templ_enum.cif

@@ -2158,7 +2158,7 @@ save_wyckoff_letter
         x
         y
         z
-         \a
+         α


This actually changes the enumeration value from \a to α thus breaking backwards compatibility with CIF1. We could probably include α as an alias of \a, but \a must be retained.

templ_attr.cif

cif_core.dic

vaitkus · 2023-06-14T12:21:22Z

cif_core.dic

@@ -24764,8 +24764,8 @@ save_atom_type_scat.gaussian_coefs

    f(s; Z) = c + Sum( a~i~ * exp(-b~i~ * s^2^), i=1:N)

-    where s = sin(theta)/lambda and theta is the diffraction angle and lambda is
-    the wavelength of the incident radiation in angstroms.
+    where s = sin(θ)/λ and θ is the diffraction angle and λ is the wavelength 


Suggested change

where s = sin(θ)/λ and θ is the diffraction angle and λ is the wavelength

where s = sin(θ)/λ and θ is the diffraction angle and λ is the wavelength

cif_core.dic

vaitkus · 2023-06-14T12:29:57Z

cif_core.dic

@@ -12129,7 +12129,7 @@ save_space_group_wyckoff.letter
 ;
    The Wyckoff letter associated with this position, as given in
    International Tables for Crystallography Volume A. The
-    enumeration value '\a' corresponds to the Greek letter 'alpha'
+    enumeration value 'α' corresponds to the Greek letter 'alpha'


This sentence does not make a lot of sense with \a replaced. However, I think that \a enumeration value should in general not be replaced (see my comment in on the temp_enum.cif file).

I agree with putting it back.

Do the quotes around the \a make it such that it isn't converted to α when it is typeset? Do enumeration values end up being typeset? or used as-is? Or do we need other trickery, like '\a' to do such a thing? @nautolycus ?

IUCr journals practice with regard to enumeration values: usually, cases where they might need to be typeset are in experimental tables or other well-defined contexts. Where appropriate, they may be reformatted on printing to satisfy house style rules (e.g. monoclinic -> Monoclinic, 'Mo K\a' -> 'Mo _K_α' ). There are style rules coded into publcif etc. for doing this.

Co-authored-by: Antanas Vaitkus <antanas.vaitkus90@gmail.com>

nautolycus · 2023-06-15T13:48:38Z

I've consulted with the IT and editorial people in Chester and there is no objection to using Unicode in the dictionary definitions.

jamesrhester · 2023-06-21T06:35:49Z

I don't think there is a need to update the dictionary date, as all changes are cosmetic, i.e. do not change the behaviour of software that relies on these definitions.

vaitkus · 2023-06-21T07:41:57Z

@jamesrhester well, a new enumeration value of 'α' was added thus be behaviour did change so the date should probably be modified.

Also, as I commented before, are we completely ok with values of _description_example.case being modified to contain Unicode since this attribute provides examples on how values should be written in actual CIF instance files. While Unicode can be used in CIF2 files (and its use is probably encouraged), the given example could not be used verbatim in CIF1.1 files.

I do not think that it is a very big deal, but it might be something we want to take into account.

jamesrhester · 2023-06-22T01:07:50Z

I guess you are right about 'α', as it was only in the template dictionary I didn't pay attention.

The extra Unicode examples are OK, as DDLm is meant to apply to any format, not just CIF1/2.

vaitkus · 2023-06-22T06:42:16Z

OK, just making sure.

rowlesmr added 13 commits June 4, 2023 09:48

replace all unambiguous markup with unicode

9a4d513

I haven't touched the update dates yet.

ambiguous unicode replacement

02b49e3

Strictly, this should be tau, but I think it means theta.

yes, it is eta

b56d5e8

Revert "attempting to fix error in layout checker"

0ebe2f8

This reverts commit 392cabc.

replacing "theta" and adjacent words

3dac14b

replace pi

d9ef2a0

replace alpha

0e0148c

replace beta

309094d

replace gamma

562d2c7

replace kappa

82a25e2

update equation alignment after unicode replacement

e872a99

missed some in templ_attr

8deef6c

This was referenced Jun 4, 2023

Unicode bug in layout checker? #414

Closed

replace markup with unicode characters COMCIFS/Powder_Dictionary#123

Merged

nautolycus closed this Jun 5, 2023

nautolycus reopened this Jun 5, 2023

Merge branch 'master' into unicode

b96f38e

vaitkus reviewed Jun 14, 2023

View reviewed changes

rowlesmr and others added 7 commits June 14, 2023 21:05

remove whitespace

78a71d2

Co-authored-by: Antanas Vaitkus <antanas.vaitkus90@gmail.com>

remove trailing whitespace

4f73737

Co-authored-by: Antanas Vaitkus <antanas.vaitkus90@gmail.com>

remove whitespace

ed05f60

Co-authored-by: Antanas Vaitkus <antanas.vaitkus90@gmail.com>

remove trailing whitespace

d9971c9

Co-authored-by: Antanas Vaitkus <antanas.vaitkus90@gmail.com>

remove trailing whitespace

ef34af5

Co-authored-by: Antanas Vaitkus <antanas.vaitkus90@gmail.com>

\a and α are equivalent enumeration values in wyckoff_letter

b284d2d

fix Non-optimal delimiters for _description_example.case

8fa4ae0

jamesrhester and others added 3 commits June 16, 2023 15:50

Merge branch 'master' into unicode

fc94ccd

Remove trailing space.

9c56690

Merge branch 'master' into unicode

ec05dd4

jamesrhester merged commit e004cb7 into COMCIFS:master Jun 21, 2023
3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace text with unicode symbols #413

Replace text with unicode symbols #413

rowlesmr commented Jun 4, 2023

rowlesmr commented Jun 4, 2023

jamesrhester commented Jun 5, 2023

rowlesmr commented Jun 5, 2023

nautolycus commented Jun 5, 2023

vaitkus left a comment

vaitkus Jun 14, 2023

vaitkus Jun 14, 2023

vaitkus Jun 14, 2023

rowlesmr Jun 14, 2023

nautolycus Jun 15, 2023

nautolycus commented Jun 15, 2023

jamesrhester commented Jun 21, 2023

vaitkus commented Jun 21, 2023

jamesrhester commented Jun 22, 2023

vaitkus commented Jun 22, 2023

@@ @@ -2158,7 +2158,7 @@ save_wyckoff_letter @@
                        x
                        y
                        z
-                       \a
+                       α

	where s = sin(θ)/λ and θ is the diffraction angle and λ is the wavelength
	where s = sin(θ)/λ and θ is the diffraction angle and λ is the wavelength

Replace text with unicode symbols #413

Replace text with unicode symbols #413

Conversation

rowlesmr commented Jun 4, 2023

rowlesmr commented Jun 4, 2023

jamesrhester commented Jun 5, 2023

rowlesmr commented Jun 5, 2023

nautolycus commented Jun 5, 2023

vaitkus left a comment

Choose a reason for hiding this comment

vaitkus Jun 14, 2023

Choose a reason for hiding this comment

vaitkus Jun 14, 2023

Choose a reason for hiding this comment

vaitkus Jun 14, 2023

Choose a reason for hiding this comment

rowlesmr Jun 14, 2023

Choose a reason for hiding this comment

nautolycus Jun 15, 2023

Choose a reason for hiding this comment

nautolycus commented Jun 15, 2023

jamesrhester commented Jun 21, 2023

vaitkus commented Jun 21, 2023

jamesrhester commented Jun 22, 2023

vaitkus commented Jun 22, 2023