Casenorm tests: enable backslash interpretation

[gmelikov]

echo must have -e argument to properly
interpret unusual symbols.

Signed-off-by: George Melikov mail@gmelikov.ru

Motivation and Context

Fix #7633

Description

Thanks to @ikozhukhov , the problem resolves with this patch.

How Has This Been Tested?

WIP, waiting for buildbot successfull finish.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the ZFS on Linux code style requirements.
• [] I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• All new and existing tests passed.
• All commit messages are properly formatted and contain Signed-off-by.

[ikozhukhov]

experiment with printf will be interest

[gmelikov]

Ah, the problem is with non-existing zlook at Linux, will try to replace it.

[behlendorf]

Thanks for looking in to this. As part of the final fix you're going to want to remove the casenorm exceptions from tests/test-runner/bin/zts-report.py too. Also if you'd like to iterate more quickly with the CI on this, I'd suggest adding TEST_ZFSTESTS_TAGS="casenorm" to your commit message so only that test group is run.

[codecov]

Codecov Report

Merging #8812 into master will increase coverage by 0.03%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #8812      +/-   ##
==========================================
+ Coverage   79.11%   79.14%   +0.03%     
==========================================
  Files         419      419              
  Lines      123704   123696       -8     
==========================================
+ Hits        97870    97902      +32     
+ Misses      25834    25794      -40

Flag	Coverage Δ
#kernel	79.9% <ø> (+0.18%)	⬆️
#user	66.46% <ø> (-0.47%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8221bcf...08ebb5c. Read the comment docs.

[rlaager]

echo -e is not POSIX compliant. Starting from bash:

$ echo -e 'F\0303\0257L\0303\0253N\0303\0204m\0303\0253'
FïLëNÄmë
$ dash
$ echo -e 'F\0303\0257L\0303\0253N\0303\0204m\0303\0253'
-e FïLëNÄmë

Is this guaranteed to run under bash?

[gmelikov]

@ikozhukhov @rlaager I've got it working via printf, just replaced lines with needed format for printf.

@behlendorf thank you, removed tests from exclusion list.

Used test -f as a cheap replacement for zlook (maybe it's not the best path). Got a problem with some tests, will dig on weekend.

[Ornias1993]

@gmelikov looks like a simple (partial) fix, any news on this? Being WIP and such...

[gmelikov]

@Ornias1993 unfortunately it's not so simple - printf does work in Linux, but some tests are failing, I think there are some problems with ksh in Linux, but I didn't find the root cause yet.

I've updated PR and excluded problematic tests from PR for now, let's see tests and maybe merge this change as is.

[behlendorf]

This could be an issue with ksh-2020.0.0, or perhaps an intentional change in behavior. When running ksh interactively it correctly interprets the unicode, but for some reason when invoked via the test suite not all \uHHHH values are interpreted (i.e \u0308).

ksh --version
  version         sh (AT&T Research) 2020.0.0
[behlendo@localhost zfs]$ ksh
$ printf '\u0066\u0069\u0308\u006c\u0065\u0308\u006e\u0061\u0308\u006d\u0065\u0308'
fïlënämë$ 

For example, a snipet from the ZTS output:

printf '\u0066\u0069\u0308\u006c\u0065\u0308\u006e\u0061\u0308\u006d\u0065\u0308'
fi\u0308le\u0308na\u0308me\u0308

Using bash -c "-printf '...'" as a workaround worked for me on Fedora 31. It's pretty ugly, but we could probably live with it as a Linux-only change. When applied all of the casenorm tests pass.

NAME_C_ORIG=$(bash -c "printf '\u0046\u00ef\u004c\u00eb\u004e\u00c4\u006d\u00eb'")
NAME_C_UPPER=$(bash -c "printf '\u0046\u00cf\u004c\u00cb\u004e\u00c4\u004d\u00cb'")
NAME_C_LOWER=$(bash -c "printf '\u0066\u00ef\u006c\u00eb\u006e\u00e4\u006d\u00eb'")
NAME_D_ORIG=$(bash -c "printf '\u0046\u0069\u0308\u004c\u0065\u0308\u004e\u0041\u0308\u006d\u0065\u0308'")
NAME_D_UPPER=$(bash -c "printf '\u0046\u0049\u0308\u004c\u0045\u0308\u004e\u0041\u0308\u004d\u0045\u0308'")
NAME_D_LOWER=$(bash -c "printf '\u0066\u0069\u0308\u006c\u0065\u0308\u006e\u0061\u0308\u006d\u0065\u0308'")

[behlendorf]

Ahh, it seems the locale is being lost from the environment. @gmelikov adding the following hunk to this PR should be enough.

diff --git a/tests/zfs-tests/tests/functional/casenorm/casenorm.cfg b/tests/zfs-tests/tests/functional/casenorm/casenorm.cfg
index e35a726e7df..f955d33cd19 100644
--- a/tests/zfs-tests/tests/functional/casenorm/casenorm.cfg
+++ b/tests/zfs-tests/tests/functional/casenorm/casenorm.cfg
@@ -17,6 +17,9 @@
 # Copyright (c) 2016 by Delphix. All rights reserved.
 #
 
+LANG=en_US.UTF-8
+LC_ALL=en_US.UTF-8
+
 NAME_C_ORIG=$(printf '\u0046\u00ef\u004c\u00eb\u004e\u00c4\u006d\u00eb')
 NAME_C_UPPER=$(printf '\u0046\u00cf\u004c\u00cb\u004e\u00c4\u004d\u00cb')
 NAME_C_LOWER=$(printf '\u0066\u00ef\u006c\u00eb\u006e\u00e4\u006d\u00eb')

[rlaager]

Instead of "en_US", does "C" or "C.UTF-8" work? If so, it might be better to use that. Otherwise, en_US.UTF-8 is probably fine.

[gmelikov]

@behlendorf thank you! Unfortunately, locale fix didn't work for buildbot, so I've used bash hack.

[gmelikov]

Bash hack didn't work either for buildbot. Interesting.

[behlendorf]

Can you try the following. @rlaager's suggestion works for me locally.

export LANG="C.UTF-8"
export LC_ALL="C.UTF-8"

Note that with this change I still do see 6 local failures like with the original PR. But everything looks to be escaped properly so those failures will need to be inspected individually.

$ ./scripts/zfs-tests.sh -T casenorm
...
Results Summary
PASS	  13
FAIL	   6

Running Time:	00:00:45
Percent passed:	68.4%
Log directory:	/var/tmp/test_results/20191118T093144

Tests with results other than PASS that are expected:
    FAIL casenorm/mixed_formd_delete (https://github.com/zfsonlinux/zfs/issues/7633)
    FAIL casenorm/mixed_formd_lookup (https://github.com/zfsonlinux/zfs/issues/7633)
    FAIL casenorm/mixed_formd_lookup_ci (https://github.com/zfsonlinux/zfs/issues/7633)
    FAIL casenorm/mixed_none_lookup_ci (https://github.com/zfsonlinux/zfs/issues/7633)
    FAIL casenorm/sensitive_formd_delete (https://github.com/zfsonlinux/zfs/issues/7633)
    FAIL casenorm/sensitive_formd_lookup (https://github.com/zfsonlinux/zfs/issues/7633)

[gmelikov]

@behlendorf ah, I thought you had all tests passed.

Added locale fix for fedora, excluded problematic tests. We may merge this PR and tackle with other tests next time.