Adding TopWikipediaSessions integration test #6311
Conversation
|
Run Java PostCommit |
| timestamp = ((BigDecimal) row.get("timestamp")).intValue(); | ||
| } catch (ClassCastException e) { | ||
| timestamp = ((Integer) row.get("timestamp")).intValue(); | ||
| } |
There was a problem hiding this comment.
Dataflow converts into BigDecimal, while Direct runner converts into Integer.
There was a problem hiding this comment.
Can you elaborate on this issue a bit please? Do we have similar issue or different types for other runners?
There was a problem hiding this comment.
@apilloud do you know why this happens on parsing of JSON files?
There was a problem hiding this comment.
If you're thinking the TableRow here is related to SQL, it isn't. This is a BigQuery datatype. The JSON parsing is being configured to use a global instance of Jackson in ParseTableRowJson, later in this file. The TableRow type is really just Map<String,Object>, so Jackson will pick a type for numbers based on global config which must not be the same on all runners. (See https://fasterxml.github.io/jackson-databind/javadoc/2.8/index.html?com/fasterxml/jackson/databind/DeserializationFeature.html for some of these config options.) One way to fix this would be to provide the actual types for json decoding rather then using Map<String,Object>.
|
r: @Ardagan |
|
@Ardagan and I had a chat offline, and I need to look into what Andrew recommended me to do. Give me a couple weeks to get to this again. |
|
@Ardagan I think I'd like to get this in for now, and keep a todo (https://issues.apache.org/jira/browse/BEAM-5390). LMK what you think |
This test should run with postcommits.
Post-Commit Tests Status (on master branch)