--noctrl option tries to replace unicode right single quotation mark

[averms]

Let's say I have a file called bob’s.txt. I run prename on it and get:

• prename -n --noctrl bob’s.txt
'bob’s.txt' would be renamed to 'bob�_s.txt'

Actually running the command would result in the file being named 62 6f 62 E2 5F 73 2E 74 78 74, which is invalid UTF-8 because of the E2 byte. I don't think this should happen.

[ap]

That won’t happen if you add -T UTF-8.

I’m very much unsure whether that is the correct answer or just a workaround. Should rename try to do that for you? Should it try to do that for you but only under specific circumstances?

[averms]

The larger issue is that --noctrl should, by default, only replace the 33 control characters defined by ASCII. This way it works with ASCII filenames and UTF-8 filenames. I'm very much a beginner at Perl so I won't try too hard to guess at what the program does, but it seems like the character class [:cntrl:] is somehow matching more than just the ASCII control characters.

Upon further reading it seems like [:cntrl:] is Unicode-aware with use feature 'unicode_strings' and rename automatically enables a feature bundle for the user's version of Perl. This breaks when the strings are not Unicode, which is the case when rename is run without any -T args.

[ap]

You’re right.

The larger issue is that --noctrl should, by default, only replace the 33 control characters defined by ASCII.

I take it you’re saying it should use Unicode-aware [:cntrl:] under -T?

[averms]

You’re right.

The larger issue is that --noctrl should, by default, only replace the 33 control characters defined by ASCII.

I take it you’re saying it should use Unicode-aware [:cntrl:] under -T?

For -T, I'm not really sure what the best path is. Currently rename does match all Unicode control characters under -T, so it would be a breaking change to restrict it to the control characters defined by ASCII.

I think the simplest solution is to no feature 'unicode_strings' if the user hasn't specified -T.

[ap]

I think the simplest solution is to no feature 'unicode_strings' if the user hasn't specified -T.

Just for --noctrl or in general? Doing that in general would affect all patterns, including those in user code passed with -e.

[averms]

I think the simplest solution is to no feature 'unicode_strings' if the user hasn't specified -T.

Just for --noctrl or in general? Doing that in general would affect all patterns, including those in user code passed with -e.

In general. As it stands enabling feature unicode_strings is incorrect when the internal encoding for a string is not utf-8. Looking at https://metacpan.org/pod/Unicode::Semantics, I imagine one could devise many examples of rename(1) outputting broken utf-8.