Export language variables

[ktf]

Old code only works if the variables were already exported.

[github-actions]

REQUEST FOR PRODUCTION RELEASES:
To request your PR to be included in production software, please add the corresponding labels called "async-" to your PR. Add the labels directly (if you have the permissions) or add a comment of the form (note that labels are separated by a ",")

+async-label <label1>, <label2>, !<label3> ...

This will add <label1> and <label2> and removes <label3>.

The following labels are available
async-2023-pbpb-apass4
async-2023-pp-apass4
async-2024-pp-apass1
async-2022-pp-apass7
async-2024-pp-cpass0
async-2024-PbPb-apass1
async-2024-ppRef-apass1
async-2024-PbPb-apass2
async-2023-PbPb-apass5

[ktf]

@davidrohr any objections to this? $'\xFF' behaves weirdly for me on mac (I did not check why though).

[davidrohr]

Hm, the current version works in the CI, and also on Anton's Mac which initially showed some issues.
Could we understand why it fails now.
I do not like such hidden fixes that can fail weirdly and silently on special code.

[ktf]

For me, given the following trivial csv:

foo,bar

it fails with:

→ GPU/GPUTracking/Definitions/Parameters/csv_to_json.sh foo.csv
+ [[ -z foo.csv ]]
+ LANG=C
+ LC_ALL=C
+ DELIM=$'\377'
+ set -o pipefail
+ sed -E $':loop\n   s/^(([^"]*"[^"]*")*[^"]*),/\\1\377/;\n   t loop' foo.csv
+ awk $'-F\377' $'BEGIN {\n     print "{"\n   } {\n     if (count == 0) {\n       for (i = 1; i <= NF; i++) {\n         names[i] = $i\n       }\n     } else if ($1 == "CORE:" || $1 == "LB:" || $1 == "PAR:") {\n       if (paramprinted) print "\\n    }"\n       else if (lineprinted) print ""\n       if (catprinted) print "  },"\n       lineprinted = 0\n       paramprinted = 0\n       catprinted = 1\n       gsub(/:$/, "", $1)\n       print "  \\""$1"\\": {";\n     } else if ($1 != "") {\n       if (lineprinted) print ""\n       if (paramprinted) print "    },"\n       lineprinted = 0\n       paramprinted = 1\n       print "    \\""$1"\\": {";\n       lineprinted = 0\n       for (i=2; i<=NF; i++) {\n         if ($i != "") {\n           gsub(/^"/, "", $i)\n           gsub(/"$/, "", $i)\n           gsub(/""/, "\\"", $i)\n           if (lineprinted) print ","\n           lineprinted = 1\n           printf("      \\"%s\\": %s", names[i], $i)\n         }\n       }\n     }\n     count++;\n   } END {\n     if (paramprinted) print "\\n    }"\n     if (catprinted) print "  }"\n     print "}"\n   }'
sed: RE error: illegal byte sequence
{
}

[davidrohr]

Hm, I don't understand. We tried at Anton's mac, which had exactly that failure before, and setting the LANG and LC_ALL fixed it.
Could you try \034 or \037 (\x1C or \x1F) ?

[ktf]

This is the proper fix. There is no guarantee that LANG and LC_ALL are actually exported in the first place and if they are not, then sed misbehaves with the unicode char. I still hold that using an unicode character as delimiter and setting the locale is probably looking for troubles.

[davidrohr]

If you could try and check if \x1C or \x1F instread of \xFF, that might be safer, since it is in the lower 128 ascii characters. It works for me on Linux but I cannot check MacOS.

[ktf]

Yes, indeed \x1F works as well. Given it's "Unit Separator" I guess that's what it was intended for...

[ktf]

Still you do need the export, otherwise other UTF8 chars in the input might confuse sed.

[davidrohr]

Yes, sure, so let's do these 2 changes and hope that then it is fixed for good.