Skip to content

Fix/json reader fixedstring v2 #448

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Nov 6, 2025
Merged

Fix/json reader fixedstring v2 #448

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1ba4ba3
Wake up ClickHouse Cloud instance before tests (#429)
BentsiLeviav Nov 4, 2025
0b788f7
fix: Handle FixedString as plain text in JSON reader for all Spark ve…
Nov 5, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions .github/workflows/cloud.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,31 @@ jobs:
distribution: zulu
java-version: 8
cache: gradle
- name: Wake up ClickHouse Cloud instance
env:
CLICKHOUSE_PASSWORD: ${{ secrets.INTEGRATIONS_TEAM_TESTS_CLOUD_PASSWORD_SMT }}
Comment on lines +48 to +49
Copy link

@windsurf-bot windsurf-bot bot Nov 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's an inconsistency in environment variable naming. The job defines CLICKHOUSE_CLOUD_PASSWORD but the wake-up script uses CLICKHOUSE_PASSWORD. Consider using the same variable name in both places or explicitly mapping between them.

run: |
echo "Waking up ClickHouse Cloud instance..."
max_attempts=3
attempt=1

while [ $attempt -le $max_attempts ]; do
echo "Attempt $attempt of $max_attempts"
if curl -sS "https://${CLICKHOUSE_CLOUD_HOST}:8443/?query=SELECT+1" \
--user "default:${CLICKHOUSE_PASSWORD}" \
--max-time 60 > /dev/null; then
echo "Instance is awake!"
break
else
if [ $attempt -eq $max_attempts ]; then
echo "Failed to wake instance after $max_attempts attempts"
exit 1
fi
echo "Retrying in 10 seconds..."
sleep 10
((attempt++))
fi
done
- run: >-
./gradlew clean cloudTest --no-daemon --refresh-dependencies
-Dspark_binary_version=${{ matrix.spark }}
Expand Down
7 changes: 6 additions & 1 deletion ...ickhouse-spark/src/main/scala/com/clickhouse/spark/read/format/ClickHouseJsonReader.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,12 @@ class ClickHouseJsonReader(
TimeUnit.SECONDS.toMicros(_instant.toEpochSecond) + TimeUnit.NANOSECONDS.toMicros(_instant.getNano())
case StringType => UTF8String.fromString(jsonNode.asText)
case DateType => LocalDate.parse(jsonNode.asText, dateFmt).toEpochDay.toInt
case BinaryType => jsonNode.binaryValue
case BinaryType if jsonNode.isTextual =>
// ClickHouse JSON format returns FixedString as plain text, not Base64
jsonNode.asText.getBytes("UTF-8")
case BinaryType =>
// True binary data is Base64 encoded in JSON format
jsonNode.binaryValue
case ArrayType(_dataType, _nullable) =>
val _structField = StructField(s"${structField.name}__array_element__", _dataType, _nullable)
new GenericArrayData(jsonNode.asScala.map(decodeValue(_, _structField)))
Expand Down
7 changes: 6 additions & 1 deletion ...ickhouse-spark/src/main/scala/com/clickhouse/spark/read/format/ClickHouseJsonReader.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,12 @@ class ClickHouseJsonReader(
TimeUnit.SECONDS.toMicros(_instant.toEpochSecond) + TimeUnit.NANOSECONDS.toMicros(_instant.getNano())
case StringType => UTF8String.fromString(jsonNode.asText)
case DateType => LocalDate.parse(jsonNode.asText, dateFmt).toEpochDay.toInt
case BinaryType => jsonNode.binaryValue
case BinaryType if jsonNode.isTextual =>
// ClickHouse JSON format returns FixedString as plain text, not Base64
jsonNode.asText.getBytes("UTF-8")
case BinaryType =>
// True binary data is Base64 encoded in JSON format
jsonNode.binaryValue
case ArrayType(_dataType, _nullable) =>
val _structField = StructField(s"${structField.name}__array_element__", _dataType, _nullable)
new GenericArrayData(jsonNode.asScala.map(decodeValue(_, _structField)))
Expand Down
7 changes: 6 additions & 1 deletion ...ickhouse-spark/src/main/scala/com/clickhouse/spark/read/format/ClickHouseJsonReader.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,12 @@ class ClickHouseJsonReader(
TimeUnit.SECONDS.toMicros(_instant.toEpochSecond) + TimeUnit.NANOSECONDS.toMicros(_instant.getNano())
case StringType => UTF8String.fromString(jsonNode.asText)
case DateType => LocalDate.parse(jsonNode.asText, dateFmt).toEpochDay.toInt
case BinaryType => jsonNode.binaryValue
case BinaryType if jsonNode.isTextual =>
// ClickHouse JSON format returns FixedString as plain text, not Base64
jsonNode.asText.getBytes("UTF-8")
case BinaryType =>
// True binary data is Base64 encoded in JSON format
jsonNode.binaryValue
case ArrayType(_dataType, _nullable) =>
val _structField = StructField(s"${structField.name}__array_element__", _dataType, _nullable)
new GenericArrayData(jsonNode.asScala.map(decodeValue(_, _structField)))
Expand Down
Loading