Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HADOOP-17945. JsonSerialization raises EOFException reading JSON data stored on google GCS #3501

Merged
merged 1 commit into from
Oct 19, 2021
Merged

HADOOP-17945. JsonSerialization raises EOFException reading JSON data stored on google GCS #3501

merged 1 commit into from
Oct 19, 2021

Conversation

steveloughran
Copy link
Contributor

(lifted from MAPREDUCE-7341)

How was this patch tested?

  • updated tests
  • rerunning s3a committer suite
  • In MAPREDUCE-7341; testing against azure abfs too
  • Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?

@steveloughran
HADOOP-17945. JsonSerialization raises EOFException reading JSON data…
d4049be 
… stored on google GCS

(lifted from MAPREDUCE-7341)

Change-Id: I0170f311cf0a33806c21c3c50e5e797fe16b0dbf
@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 1m 17s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 38m 0s trunk passed
+1 💚 compile 27m 20s trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 compile 23m 38s trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 checkstyle 1m 9s trunk passed
+1 💚 mvnsite 1m 48s trunk passed
+1 💚 javadoc 1m 13s trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javadoc 1m 47s trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 spotbugs 2m 57s trunk passed
+1 💚 shadedclient 27m 38s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 1m 11s the patch passed
+1 💚 compile 27m 30s the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javac 27m 30s the patch passed
+1 💚 compile 24m 45s the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 javac 24m 45s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 1m 5s the patch passed
+1 💚 mvnsite 1m 42s the patch passed
+1 💚 javadoc 1m 7s the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javadoc 1m 47s the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 spotbugs 3m 5s the patch passed
+1 💚 shadedclient 27m 53s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 19m 22s hadoop-common in the patch passed.
+1 💚 asflicense 0m 54s The patch does not generate ASF License warnings.
236m 25s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3501/1/artifact/out/Dockerfile
GITHUB PR #3501
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname Linux 0faf8f3c9113 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / d4049be
Default Java Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3501/1/testReport/
Max. process+thread count 2898 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3501/1/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@steveloughran
Copy link
Contributor Author

@mehakmeet & @mukund-thakur could you take a quick look @ this?

mukund-thakur
mukund-thakur reviewed Sep 30, 2021
View reviewed changes
Copy link
Contributor

@mukund-thakur mukund-thakur left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added few questions.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/JsonSerialization.java Show resolved Hide resolved
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/JsonSerialization.java Show resolved Hide resolved
...common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestJsonSerialization.java
@@ -181,5 +197,23 @@ public void testFileSystemEmptyPath() throws Throwable {
}
}

/**
* 0 byte file through the load(path, status) API will fail with an
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we add a test with 0 byte file but not passing the fileStatus.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

probably

@mukund-thakur
Copy link
Contributor

Just one test pending here then. Post that we can merge.

@steveloughran
Copy link
Contributor Author

i'm not in a setup to write tests. i don't have a work laptop and having just tried to check it out on my windows laptop, the paths for yarn timeline server are too long for the windows FS

I'm not in a position to even check out the source, let alone compile it.

can we take it as is?

mukund-thakur
mukund-thakur approved these changes Oct 19, 2021
View reviewed changes
Copy link
Contributor

@mukund-thakur mukund-thakur left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 LGTM

@mukund-thakur mukund-thakur merged commit 2194b97 into apache:trunk Oct 19, 2021
mukund-thakur pushed a commit that referenced this pull request Oct 19, 2021
@steveloughran @mukund-thakur
HADOOP-17945. JsonSerialization raises EOFException reading JSON data…
b8f3e54 
… stored on google GCS (#3501)


Contributed By: Steve Loughran
HarshitGupta11 pushed a commit to HarshitGupta11/hadoop that referenced this pull request Nov 28, 2022
@steveloughran
HADOOP-17945. JsonSerialization raises EOFException reading JSON data…
0d43893 
… stored on google GCS (apache#3501)


Contributed By: Steve Loughran
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mukund-thakur mukund-thakur mukund-thakur approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@steveloughran @hadoop-yetus @mukund-thakur