Skip to content

Strip and safer conversion of values to float to avoid error see HEA-615 #160

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Sep 19, 2025
Merged

Strip and safer conversion of values to float to avoid error see HEA-615 #160

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1503e5d
Strip and safer conversion of values to float to avoid error see HEA-615
girumb Sep 19, 2025
eca6ea8
Better handling converting percentage_of_households - see HEA-615
rhunwicks Sep 19, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 35 additions & 11 deletions pipelines/assets/wealth_characteristic.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -317,26 +317,50 @@ def wealth_characteristic_instances(

wealth_group_characteristic_value["reference_type"] = reference_type

# The percentagme of households should be stored as a number between 1 and 100,
# The percentage of households should be stored as a number between 1 and 100,
# but may be stored in the BSS (particularly in the summary column) as a
# decimal fraction between 0 and 1, so correct those values
if (
wealth_group_characteristic_value["wealth_characteristic_id"] == "percentage of households"
and value != ""
and float(value) < 1
):
value = float(value) * 100
try:
if (
wealth_group_characteristic_value["wealth_characteristic_id"]
== "percentage of households"
and str(value).strip()
and float(value) < 1
):
value = float(value) * 100
except Exception as e:
raise ValueError(
"Error in %s converting percentage of households value '%s' to float from 'WB'!%s%s"
% (partition_key, value, column, row)
) from e

wealth_group_characteristic_value["value"] = value

# If this is the summary, then also save the min and max values
if reference_type == WealthGroupCharacteristicValue.CharacteristicReference.SUMMARY:
min_value = df.loc[row, df.columns[-2]]
if min_value != "" and float(min_value) < 1:
min_value = float(min_value) * 100
max_value = df.loc[row, df.columns[-1]]
if max_value != "" and float(max_value) < 1:
max_value = float(max_value) * 100
# Convert min/max percentage of households values from decimal fractions to percentages
if (
wealth_group_characteristic_value["wealth_characteristic_id"]
== "percentage of households"
):
try:
if str(min_value).strip() and float(min_value) < 1:
min_value = float(min_value) * 100
except Exception as e:
raise ValueError(
"Error in %s converting percentage of households value '%s' to float from 'WB'!%s%s"
% (partition_key, min_value, df.columns[-2], row)
) from e
try:
if str(max_value).strip() and float(max_value) < 1:
max_value = float(max_value) * 100
except Exception as e:
raise ValueError(
"Error in %s converting percentage of households value '%s' to float from 'WB'!%s%s"
% (partition_key, max_value, df.columns[-1], row)
) from e
wealth_group_characteristic_value["min_value"] = min_value
wealth_group_characteristic_value["max_value"] = max_value

Expand Down