AltFileInputStream.java replace FileInputStream.java in apache/hadoop/HDFS #42
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 30 commits
July 19, 2015 17:50
…y,cause the act of change can damage other part of the HDFS.For example,When I test my code using a Single Node (psedo-distributed) Cluster."Failed to load an FSImage file." will happen when I start HDFS Daemons.At start,I replace many FileInputStream which happend as an arg or constructor to NewFileInputStream,but it seems like wrong.So,I have to do this in another way.
think,although there is a man named Alan Bateman from https://bugs.openjdk.java.net/browse/JDK-8080225 suggest that.But test shows it is not good.Some problem may happen.And these test consume so long time.Every time I change the source code,I need to build the whole project (maybe it is not needed).But I install the new version hadoop on my computer.So,build the whole project is needed.Maybe should have a good way to do it I think.
from this commit.Important question I think whether AltFileInputStream could convert to FileInputStream safely.I define a frame plan to do it.But I don't know is this correct for the problem ? In HDFS code,compulsory conversion to FileInputStream is happend everywhere.
…am is the subclass of the InputStream.In the previous version of the HDFS,the convert to FileInputStream I see is safety cause these method return InputStream which is the superclass of the FileInputStream.In my version of HDFS,InputStream is also the superclass of the AltFileInputStream.So,AltFileInputStream is also a InputStream just like the FileInputStream is a InputStream too.So,I think it is safety.Everyone agree ? If not,please give your opinion and tell me What's wrong with that ? thank you.
cause the channel to be closed.
old patch.
shanthoosh
pushed a commit
to shanthoosh/hadoop
that referenced
this pull request
Oct 15, 2019
* Wrote a unit/integration test to simulate a stuck connection when reading binaries for the job. Other misc. changes: - Moved some debug log messages to be info for better debugging. Author: vjagadish1989 <jvenkatr@linkedin.com> Reviewers: jmakes,nickpan47 Closes apache#42 from vjagadish/http-fs
saxenapranav
pushed a commit
to saxenapranav/hadoop
that referenced
this pull request
Apr 28, 2023
Added Import Assume
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
A brief description
Long Stop-The-World GC pauses due to Final Reference processing are observed.
So, Where are those Final Reference come from ?
1 :
Finalizer2 :
FileInputStreamHow to solve this problem ?
Here is the detailed description,and I give a solution on this.
https://issues.apache.org/jira/browse/HDFS-8562
FileInputStream have a method of finalize , and it can cause GC pause for a long time.In our test,G1 as our GC. So,in AltFileInputStream , no finalize. A new design for a inputstream use in windows and non-windows.