[SPARK-28151][SQL] Fix MsSqlServerDialect Byte/Short/Float type mappings ( DRAFT) #24969

shivsood · 2019-06-25T17:57:52Z

Abandoning this PR. Will create new one that have specific fixes

PR for Float Type and ShortType Fix
PR for ByteType Fix.

Fix for ByteType, ShortType and FloatTypes are not correctly mapped for read/write of SQLServer tables
###ByteType issue
Writing dataframe with column type BYTETYPE fails when using JDBC connector for SQL Server. Append and Read of tables also fail. The problem is due

(Write path) Incorrect mapping of BYTETYPE in getCommonJDBCType() in jdbcutils.scala where BYTETYPE gets mapped to BYTE text. It should be mapped to TINYINT
case ByteType => Option(JdbcType("BYTE", java.sql.Types.TINYINT))

In getCatalystType() ( JDBC to Catalyst type mapping) TINYINT is mapped to INTEGER, while it should be mapped to BYTETYPE. Mapping to integer is ok from the point of view of upcasting, but will lead to 4 byte allocation rather than 1 byte for BYTETYPE.

(read path) Read path ends up calling makeGetter(dt: DataType, metadata: Metadata). The function sets the value in RDD row. The value is set per the data type. Here there is no mapping for BYTETYPE and thus results will result in an error when getCatalystType() is fixed.

Note : These issues were found when reading/writing with SQLServer. Will be submitting a PR soon to fix these mappings in MSSQLServerDialect.

Error seen when writing table

(JDBC Write failed,com.microsoft.sqlserver.jdbc.SQLServerException: Column, parameter, or variable #2: Cannot find data type BYTE.)
com.microsoft.sqlserver.jdbc.SQLServerException: Column, parameter, or variable #2: Cannot find data type BYTE.
com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:254)
com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServerStatement.java:1608)
com.microsoft.sqlserver.jdbc.SQLServerStatement.doExecuteStatement(SQLServerStatement.java:859)
..

###ShortType and FloatType issue
ShortType and FloatTypes are not correctly mapped to right JDBC types when using JDBC connector. This results in tables and spark data frame being created with unintended types.

Some example issue

Write from df with column type results in a SQL table of with column type as INTEGER as opposed to SMALLINT. Thus a larger table that expected.
read results in a dataframe with type INTEGER as opposed to ShortType

FloatTypes have a issue with read path. In the write path Spark data type 'FloatType' is correctly mapped to JDBC equivalent data type 'Real'. But in the read path when JDBC data types need to be converted to Catalyst data types ( getCatalystType) 'Real' gets incorrectly gets mapped to 'DoubleType' rather than 'FloatType'.

What changes were proposed in this pull request?

(MsSqlServerDialect.scala ) BYTETYPE is mapped to "TINYINT", SHORTTYPE is mapped to small int
(JdbcUtils.scala) reading ByteTYPE is added to makeGetter to enable reading after ByteType is translated to TINYINT.

How was this patch tested?

Unit test - Added and passed.
Integration Test - Tested end to end with SQLServer using JDBC connector.

(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)

Please review https://spark.apache.org/contributing.html before opening a pull request.

…ta sources.Relevant unit tests are added. Issues: Writing dataframe with column type BYTETYPE fails when using JDBC connector for SQL Server. Append and Read of tables also fail. The problem is due to 1. (Write path) Incorrect mapping of BYTETYPE in getCommonJDBCType() in jdbcutils.scala where BYTETYPE gets mapped to BYTE text. It should be mapped to TINYINT case ByteType => Option(JdbcType("BYTE", java.sql.Types.TINYINT)) In getCatalystType() ( JDBC to Catalyst type mapping) TINYINT is mapped to INTEGER, while it should be mapped to BYTETYPE. Mapping to integer is ok from the point of view of upcasting, but will lead to 4 byte allocation rather than 1 byte for BYTETYPE. 2. (Read path) Read path ends up calling makeGetter(dt: DataType, metadata: Metadata). The function sets the value in RDD row. The value is set per the data type. Here there is no mapping for BYTETYPE and thus results will result in an error when getCatalystType() is fixed. Note : These issues were found when reading/writing with SQLServer. Error seen when writing table (JDBC Write failed,com.microsoft.sqlserver.jdbc.SQLServerException: Column, parameter, or variable apache#2: Cannot find data type BYTE.) com.microsoft.sqlserver.jdbc.SQLServerException: Column, parameter, or variable apache#2: Cannot find data type BYTE. com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:254) com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServerStatement.java:1608) com.microsoft.sqlserver.jdbc.SQLServerStatement.doExecuteStatement(SQLServerStatement.java:859) ..

ShortType and FloatTypes are not correctly mapped to right JDBC types when using JDBC connector. This results in tables and spark data frame being created with unintended types. Some example issue Write from df with column type results in a SQL table of with column type as INTEGER as opposed to SMALLINT. Thus a larger table that expected. Read results in a dataframe with type INTEGER as opposed to ShortType FloatTypes have a issue with read path. In the write path Spark data type 'FloatType' is correctly mapped to JDBC equivalent data type 'Real'. But in the read path when JDBC data types need to be converted to Catalyst data types ( getCatalystType) 'Real' gets incorrectly gets mapped to 'DoubleType' rather than 'FloatType'.

sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala

srowen

I don't know SQL Server, so can't really evaluate it, but seems like a plausible change.

sql/core/src/main/scala/org/apache/spark/sql/jdbc/MsSqlServerDialect.scala

srowen · 2019-06-26T14:44:35Z

sql/core/src/main/scala/org/apache/spark/sql/jdbc/MsSqlServerDialect.scala

@@ -39,6 +45,8 @@ private object MsSqlServerDialect extends JdbcDialect {
    case StringType => Some(JdbcType("NVARCHAR(MAX)", java.sql.Types.NVARCHAR))
    case BooleanType => Some(JdbcType("BIT", java.sql.Types.BIT))
    case BinaryType => Some(JdbcType("VARBINARY(MAX)", java.sql.Types.VARBINARY))
+    case ByteType => Option(JdbcType("TINYINT", java.sql.Types.TINYINT))
+    case ShortType => Option(JdbcType("SMALLINT", java.sql.Types.SMALLINT))


Does FloatType need to map back to REAL or does that already happen?

Yes, FloatType is correctly mapped to REAL in def getCommonJDBCType(dt: DataType)

…NT and FLOAT in MsSqlServerIntegrationSuite.scala. 2. code cleanup in MsSqlServerDialect.scala 3. Fixed issue in JdbcUtils where tinyInt was being written as Int and failing the 'basic write' test cases in IntegrationSuite.

dongjoon-hyun · 2019-07-05T08:04:33Z

ok to test

SparkQA · 2019-07-05T08:08:49Z

Test build #107268 has finished for PR 24969 at commit b874527.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun

Hi, @shivsood . Thank you for your contribution.
Could you run dev/scalastyle in your environment and fix all errors?

Currently, this PR has the followings.

========================================================================
Running Scala style checks
========================================================================
[info] Checking Scala style using SBT with these profiles:  -Phadoop-2.7 -Pkubernetes -Phive-thriftserver -Phadoop-cloud -Pkinesis-asl -Pyarn -Pspark-ganglia-lgpl -Phive -Pmesos
Scalastyle checks failed at following occurrences:
[error] /home/jenkins/workspace/SparkPullRequestBuilder@2/sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala:909:68: No space after token ,
[error] /home/jenkins/workspace/SparkPullRequestBuilder@2/sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala:909:78: No space after token ,
[error] /home/jenkins/workspace/SparkPullRequestBuilder@2/sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala:909:80: No space after token ,
[error] /home/jenkins/workspace/SparkPullRequestBuilder@2/sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala:911:69: No space after token ,
[error] /home/jenkins/workspace/SparkPullRequestBuilder@2/sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala:911:80: No space after token ,
[error] /home/jenkins/workspace/SparkPullRequestBuilder@2/sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala:911:82: No space after token ,
[error] /home/jenkins/workspace/SparkPullRequestBuilder@2/sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala:913:65: No space after token ,
[error] /home/jenkins/workspace/SparkPullRequestBuilder@2/sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala:913:72: No space after token ,
[error] /home/jenkins/workspace/SparkPullRequestBuilder@2/sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala:913:74: No space after token ,
[error] Total time: 27 s, completed Jul 5, 2019 1:08:48 AM

sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala

SparkQA · 2019-07-08T17:57:10Z

Test build #107364 has finished for PR 24969 at commit 8a7d960.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

shivsood · 2019-07-08T18:41:19Z

Test build #107364 has finished for PR 24969 at commit 8a7d960.
* This patch **fails to build**.

* This patch merges cleanly.

* This patch adds no public classes.

Resolved this. I am not using this exact command below to build locally and thus did not see that. Have to see the right/strict way to build that shows all all error locally.

[info] Done packaging.
java.lang.RuntimeException: 1 fatal warnings
at scala.sys.package$.error(package.scala:27)
at SparkBuild$$anonfun$sharedSettings$21.apply(SparkBuild.scala:315)
at SparkBuild$$anonfun$sharedSettings$21.apply(SparkBuild.scala:284)
at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
at sbt.std.Transform$$anon$4.work(System.scala:63)
at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
at sbt.Execute.work(Execute.scala:237)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[error] (sql/compile:compile) 1 fatal warnings
[error] Total time: 161 s, completed Jul 8, 2019 10:57:09 AM
[error] running /home/jenkins/workspace/SparkPullRequestBuilder/build/sbt -Phadoop-2.7 -Pkubernetes -Phive-thriftserver -Phadoop-cloud -Pkinesis-asl -Pyarn -Pspark-ganglia-lgpl -Phive -Pmesos test:package streaming-kinesis-asl-assembly/assembly ; received return code 1
Attempting to post to Github...

Post successful.
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error?
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107364/
Test FAILed.
Finished: FAILURE

SparkQA · 2019-07-08T19:53:33Z

Test build #4816 has finished for PR 24969 at commit 8a7d960.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

…hould have seen this as a build failure earlier.

SparkQA · 2019-07-08T23:02:48Z

Test build #107370 has finished for PR 24969 at commit 6bf6e3d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala

sql/core/src/main/scala/org/apache/spark/sql/jdbc/MsSqlServerDialect.scala

sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala

dongjoon-hyun · 2019-07-09T04:27:14Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala

@@ -550,7 +550,7 @@ object JdbcUtils extends Logging {

    case ByteType =>
      (stmt: PreparedStatement, row: Row, pos: Int) =>
-        stmt.setInt(pos + 1, row.getByte(pos))
+        stmt.setByte(pos + 1, row.getByte(pos))


This is orthogonal to MySqlServerDialect. Can we proceed this PR without this line?

Or, we can proceed this first as another PR with a proper regression test.

@dongjoon-hyun JDBCUtils.scala fix is required for completeness of the ByteType Fix. Without this fix Integrations test in "external/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/MsSqlServerIntegrationSuite.scala" will fail.

I dont think the "MsSqlServerIntegrationSuite.scala" integration tests are running in CI/CD. I was running these locally following a review suggestion. Is that's true, i can still remove this fix and submit this as part of another PR.

Then, you need to follow the second option I gave. Please make another PR. We need to review that first before this PR.

Or, we can proceed this first as another PR with a proper regression test.

@dongjoon-hyun Can u please elaborate on proper regression test. I have test cases added/fixed for ByteType in JDBCSuite.scala and MsSqlServerIntegrationSuite.scala. Are you expecting some other test? Can u please point me to type of test.

Another way to cleanly split this PR would be as follows.

Remove ByteType fix from this PR and jdbcutils.scala fix also get removed. This PR will only have fix for FloatType and Short.

2nd PR with complete ByteType fix with more elaborate integration test.

I prefer this for better seperation of concern in each of the PRs , while still retaining the completeness. Let me know what you think.

Thanks for your patience and very helpful guidance.

What I meant is like your 2nd option.

2nd PR with complete ByteType fix with more elaborate integration test.

Here, the proper regression test means a test causing failure for the other Dialects. (e.g. PostgreSQL/MySQL/...) because JdbcUtils is shared across Dialects.

@dongjoon-hyun Would create 2 PRs then. PR1 ( this PR) i will remove ByteType and JDBCUtilis fix. Will create PR2 for ByteType and JDBCUtilis fix with full integration test. PR2 will take time as i have to investigate how to test for other dialects.

Yep. Please try as you want. Then, ping me later when the PR is ready.

dongjoon-hyun · 2019-07-09T04:30:09Z

@shivsood . JdbcUtils.scala change should be handled in a separate PR.

SparkQA · 2019-07-09T19:23:01Z

Test build #107412 has finished for PR 24969 at commit f2700eb.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2019-07-12T13:19:03Z

@shivsood do you want to update this PR accordingly?

shivsood · 2019-07-12T19:49:30Z

@shivsood do you want to update this PR accordingly?
'll abandon this PR and start with a new/clean one. Have update the summary.

srowen · 2019-07-15T14:33:44Z

@shivsood did you open a new PR?
This one looked pretty fine overall minus moving out some JdbcUtils changes. I don't want to lose your improvement.

dongjoon-hyun · 2019-07-15T16:29:03Z

Yes. We are working on #25146 .

shivsood · 2019-07-15T16:39:33Z

@srowen Working on #25146 for FloatType and ShortType change. Will raise a PR for ByteType change this week ( It need more testing and i am finding my way thru adding E2E regression test)

nkronenfeld · 2020-04-06T21:27:23Z

Was a PR ever submitted on the Byte/TinyInt problem? If not, does anyone know why? We are still getting burned by this.

frgomes · 2020-07-07T11:13:02Z

@dongjoon-hyun @shivsood:: I'm getting this exception below which makes me consider that maybe the fix was not applied??

com.microsoft.sqlserver.jdbc.SQLServerException: Column, parameter, or variable #7: Cannot find data type BYTE.

Was the fix for ByteType ever submitted/applied? To which branch, please?

Thanks

srowen · 2020-07-07T11:17:10Z

#25146 ? see above

frgomes · 2020-07-07T11:49:22Z

@srowen :: #25146 only covers ShortType (smallint) and FloatType (real). ByteType (tinyint) seems to be covered here and only here. However, I'm facing an issue which makes me believe that the fix somehow is not implemented in the runtime I'm using.

frgomes · 2020-07-07T12:06:46Z

@srowen : Feedback: As suspected, the issue was between the keyboard and the chair. Maybe I was not casting properly, or maybe a typo or maybe some equality test somewhere... whatever... when I cast like this .cast(pyspark.sql.types.ByteType()), it works.

shivsood added 2 commits June 25, 2019 00:00

shivsood changed the title ~~[SPARK-28151] ByteType is not correctly mapped for read/write of SQLServer tables~~ [SPARK-28151] ByteType, ShortType and FloatTypes are not correctly mapped for read/write of SQLServer tables Jun 25, 2019

wangyum reviewed Jun 26, 2019

View reviewed changes

sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala Outdated Show resolved Hide resolved

srowen reviewed Jun 26, 2019

View reviewed changes

dongjoon-hyun added the SQL label Jun 26, 2019

dongjoon-hyun changed the title ~~[SPARK-28151] ByteType, ShortType and FloatTypes are not correctly mapped for read/write of SQLServer tables~~ [SPARK-28151][SQL] ByteType, ShortType and FloatTypes are not correctly mapped for read/write of SQLServer tables Jul 5, 2019

dongjoon-hyun requested changes Jul 6, 2019

View reviewed changes

dongjoon-hyun reviewed Jul 6, 2019

View reviewed changes

sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala Outdated Show resolved Hide resolved

dongjoon-hyun reviewed Jul 6, 2019

View reviewed changes

sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala Outdated Show resolved Hide resolved

fixed styling issues from dev/stylecheck run

8a7d960

Fixing issues in orignal commit. Change was not in the right place. S…

6bf6e3d

…hould have seen this as a build failure earlier.

dongjoon-hyun reviewed Jul 9, 2019

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala Outdated Show resolved Hide resolved

dongjoon-hyun reviewed Jul 9, 2019

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/jdbc/MsSqlServerDialect.scala Outdated Show resolved Hide resolved

dongjoon-hyun reviewed Jul 9, 2019

View reviewed changes

sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala Outdated Show resolved Hide resolved

dongjoon-hyun reviewed Jul 9, 2019

View reviewed changes

sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala Outdated Show resolved Hide resolved

dongjoon-hyun reviewed Jul 9, 2019

View reviewed changes

sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala Outdated Show resolved Hide resolved

dongjoon-hyun changed the title ~~[SPARK-28151][SQL] ByteType, ShortType and FloatTypes are not correctly mapped for read/write of SQLServer tables~~ [SPARK-28151][SQL] Fix MsSqlServerDialect Byte/Short/Float type mappings Jul 9, 2019

dongjoon-hyun reviewed Jul 9, 2019

View reviewed changes

fixing styling issues

f2700eb

shivsood closed this Jul 12, 2019

shivsood reopened this Jul 12, 2019

shivsood changed the title ~~[SPARK-28151][SQL] Fix MsSqlServerDialect Byte/Short/Float type mappings~~ [SPARK-28151][SQL] Fix MsSqlServerDialect Byte/Short/Float type mappings ( DRAFT) Jul 12, 2019

shivsood closed this Jul 12, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-28151][SQL] Fix MsSqlServerDialect Byte/Short/Float type mappings ( DRAFT) #24969

[SPARK-28151][SQL] Fix MsSqlServerDialect Byte/Short/Float type mappings ( DRAFT) #24969

shivsood commented Jun 25, 2019 •

edited

Loading

srowen left a comment

srowen Jun 26, 2019

shivsood Jul 1, 2019

dongjoon-hyun commented Jul 5, 2019

SparkQA commented Jul 5, 2019

dongjoon-hyun left a comment

SparkQA commented Jul 8, 2019

shivsood commented Jul 8, 2019 •

edited

Loading

SparkQA commented Jul 8, 2019

SparkQA commented Jul 8, 2019

dongjoon-hyun Jul 9, 2019 •

edited

Loading

dongjoon-hyun Jul 9, 2019

shivsood Jul 9, 2019

dongjoon-hyun Jul 9, 2019 •

edited

Loading

shivsood Jul 9, 2019

dongjoon-hyun Jul 9, 2019

shivsood Jul 10, 2019

dongjoon-hyun Jul 10, 2019

dongjoon-hyun commented Jul 9, 2019

SparkQA commented Jul 9, 2019

srowen commented Jul 12, 2019

shivsood commented Jul 12, 2019

srowen commented Jul 15, 2019

dongjoon-hyun commented Jul 15, 2019

shivsood commented Jul 15, 2019

nkronenfeld commented Apr 6, 2020

frgomes commented Jul 7, 2020 •

edited

Loading

srowen commented Jul 7, 2020

frgomes commented Jul 7, 2020 •

edited

Loading

frgomes commented Jul 7, 2020

[SPARK-28151][SQL] Fix MsSqlServerDialect Byte/Short/Float type mappings ( DRAFT) #24969

[SPARK-28151][SQL] Fix MsSqlServerDialect Byte/Short/Float type mappings ( DRAFT) #24969

Conversation

shivsood commented Jun 25, 2019 • edited Loading

What changes were proposed in this pull request?

How was this patch tested?

srowen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dongjoon-hyun commented Jul 5, 2019

SparkQA commented Jul 5, 2019

dongjoon-hyun left a comment

Choose a reason for hiding this comment

SparkQA commented Jul 8, 2019

shivsood commented Jul 8, 2019 • edited Loading

SparkQA commented Jul 8, 2019

SparkQA commented Jul 8, 2019

dongjoon-hyun Jul 9, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dongjoon-hyun Jul 9, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dongjoon-hyun commented Jul 9, 2019

SparkQA commented Jul 9, 2019

srowen commented Jul 12, 2019

shivsood commented Jul 12, 2019

srowen commented Jul 15, 2019

dongjoon-hyun commented Jul 15, 2019

shivsood commented Jul 15, 2019

nkronenfeld commented Apr 6, 2020

frgomes commented Jul 7, 2020 • edited Loading

srowen commented Jul 7, 2020

frgomes commented Jul 7, 2020 • edited Loading

frgomes commented Jul 7, 2020

shivsood commented Jun 25, 2019 •

edited

Loading

shivsood commented Jul 8, 2019 •

edited

Loading

dongjoon-hyun Jul 9, 2019 •

edited

Loading

dongjoon-hyun Jul 9, 2019 •

edited

Loading

frgomes commented Jul 7, 2020 •

edited

Loading

frgomes commented Jul 7, 2020 •

edited

Loading