1.50: encoding rule is ignored when csv rules file is input file

[paul-dest]

hledger 1.50.1, linux-x86_64 on Ubuntu 22.04.5 LTS

Trying to
hledger import csv.comdirectGiro.rules --dry
results in

hledger: Error: ././import/comdirectGiro/umsaetze_XXXXXXXXXX_20250924-0909.csv: hGetContents: invalid argument (cannot decode byte sequence starting from 228)
Some text could not be decoded with the system text encoding, UTF-8.
Please either convert the text to this encoding,
or configure a system locale which can decode this text.

Same with hledger -f csv.comdirectGiro.rules print.

In contrast to that, doing the same with version 1.43.2
~/hledger-linux-x64/hledger import csv.comdirectGiro.rules --dry
results as expected in

hledger: Error: could not parse "Alter Kontostand" as a date using date format "%d.%m.%Y"
record: Alter Kontostand,1.665,70 EUR,
the date rule is:   %Buchungstag
the date-format is: %d.%m.%Y
you may need to change your date rule, change your date-format rule, or change your skip rule
for m/d/y or d/m/y dates, use date-format %-m/%-d/%Y or date-format %-d/%-m/%Y

The rules file sets the encoding explicitly to iso-8859-15. Contents of csv.comdirectGiro.rules:

source ./import/comdirectGiro/umsaetze_XXXXXXXXXX*.csv
encoding	iso-8859-15
separator	;
skip		3
date-format	%d.%m.%Y
decimal-mark	,
fields		Buchungstag,Valuta,Vorgang,Buchungstext,Umsatz
date		%Buchungstag
account1	assets:eur:comdirect Giro
amount		%Umsatz EUR

if
%Buchungstext ^Auftraggeber: (.*) Buchungstext: .*
%Buchungstext ^Empfänger: (.*) Buchungstext: .*
%Buchungstext ^Buchungstext: (.*) Karte Nr..*
  description \1

if %Valuta ^(..).(..).(....)$
  comment1	date: \3-\2-\1\nvorgang: %Vorgang\nwhat: %Buchungstext

locale

LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC=de_DE.UTF-8
LC_TIME=de_DE.UTF-8
LC_COLLATE="en_US.UTF-8"
LC_MONETARY=de_DE.UTF-8
LC_MESSAGES="en_US.UTF-8"
LC_PAPER=de_DE.UTF-8
LC_NAME=de_DE.UTF-8
LC_ADDRESS=de_DE.UTF-8
LC_TELEPHONE=de_DE.UTF-8
LC_MEASUREMENT=de_DE.UTF-8
LC_IDENTIFICATION=de_DE.UTF-8
LC_ALL=

The error message of 1.50.1 "cannot decode byte sequence starting from 228" looks misleading to me as in file umsaetze_XXXXXXXXXX_20250924-0909.csv, there are only ASCII characters on and around position 228. The file contains some German characters äÜü, but on positions way before and way after position 228.

Furthermore, it is not understandable why hledger seemingly tries to decode the file as UTF-8 ("Some text could not be decoded with the system text encoding, UTF-8.") as the rules files sets the encoding explicitly to iso-8859-15.

[simonmichael]

Thanks for the report. It's unexpected for those hledger versions to handle this differently, and I can't reproduce it here. What does file umsaetze_XXXXXXXXXX_20250924-0909.csv say ? Also (removing variables), does hledger print -f import/comdirectGiro/umsaetze_XXXXXXXXXX_20250924-0909.csv --rules csv.comdirectGiro.rules behave the same way ?

[simonmichael]

Furthermore, it is not understandable why hledger seemingly tries to decode the file as UTF-8 ("Some text could not be decoded with the system text encoding, UTF-8.") as the rules files sets the encoding explicitly to iso-8859-15.

You're right, that error message is probably wrong and needs to be modified to allow for a CSV encoding rule.

[simonmichael]

I suspect file import/comdirectGiro/umsaetze_XXXXXXXXXX*.csv may show some files with a different encoding. In your examples hledger will be picking the latest-modified one.

[paul-dest]

What does file umsaetze_XXXXXXXXXX_20250924-0909.csv say ? Also (removing variables), does hledger print -f import/comdirectGiro/umsaetze_XXXXXXXXXX_20250924-0909.csv --rules csv.comdirectGiro.rules behave the same way ?

Thank you for having a first look at the problem.

The behavior is very confusing. Loading the CSV files by directly pointing at them (=surpassing the glob in the rules file) does not report any errors and works fine. The following commands, 1 and 2 both, do absolutely fine.
1:
hledger --rules csv.comdirectGiro.rules print -f ././import/comdirectGiro/umsaetze_XXXXXXXXXX_20250924-0909.csv

2:
hledger --rules csv.comdirectGiro.rules import ././import/comdirectGiro/umsaetze_XXXXXXXXXX_20250924-0909.csv --dry

while the earlier reported hledger import csv.comdirectGiro.rules --dry still produces the error message as given in my first post.

I rmed all files from ././import/comdirectGiro but the ././import/comdirectGiro/umsaetze_XXXXXXXXXX_20250924-0909.csv. Same behavior.

I changed the first line of the rules file to point directly to the file:
source ./import/comdirectGiro/umsaetze_XXXXXXXXXX_20250924-0909.csv. Same behavior.

The error seems to arise only if the file argument of the import command is a rules file. Direct pointing at the CSV file does not produce any erroneous or unexpected behavior.

[paul-dest]

Oh, forgot:
file umsaetze_XXXXXXXXXX_20250924-0909.csv says
umsaetze_XXXXXXXXXX_20250924-0909.csv: ISO-8859 text, with CRLF line terminators

[simonmichael]

Reproduced! As you say, it's ignoring the encoding rule when the rules file is the input file. Good catch!

skip 1
fields date, amount, description
source ./a.iso-8859-15.csv
encoding iso-8859-15

$ hledger print -f a.iso-8859-15.csv         # works
$ hledger print -f a.iso-8859-15.csv.rules   # fails

The workaround for now is to use the first form.

[simonmichael]

And yes it is a regression in 1.50. Claim your https://hledger.org/REGRESSIONS.html#regression-finder-bounty when ready!