HDFS-16130. Implement CREATE File with FGL #3205

prasad-acit · 2021-07-14T18:57:27Z

HDFS-16130. Implement CREATE File with FGL

…y Xing Lin. (apache#3197)

xinglin

new to HDFS but I tried my best to write some comments. Please bear with me, if some of the comments look naive to you.

...oject/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java

shvachko

General suggestion here.

Seems there is a fair amount of refactoring going on in the patch, like adding createMissingDirs(). We can do such refactoring on trunk, and then this change will be smaller, which will make fgl branch maintenance simpler.
It would be good to have a test one can run to verify the change works. Not necessarily a new test. Could be an existing one.

...s-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirMkdirOp.java

...oject/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java

hadoop-yetus · 2021-07-22T02:16:20Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 50s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ fgl Compile Tests _
+1 💚	mvninstall	34m 41s		fgl passed
+1 💚	compile	1m 37s		fgl passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	1m 26s		fgl passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	1m 3s		fgl passed
+1 💚	mvnsite	1m 25s		fgl passed
+1 💚	javadoc	1m 2s		fgl passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 27s		fgl passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
-1 ❌	spotbugs	3m 44s	/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html	hadoop-hdfs-project/hadoop-hdfs in fgl has 2 extant spotbugs warnings.
-1 ❌	shadedclient	20m 23s		branch has errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 26s		the patch passed
+1 💚	compile	1m 34s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	1m 34s		the patch passed
+1 💚	compile	1m 22s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	1m 22s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 59s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project/hadoop-hdfs: The patch generated 28 new + 16 unchanged - 0 fixed = 44 total (was 16)
+1 💚	mvnsite	1m 26s		the patch passed
+1 💚	javadoc	1m 2s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 32s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	3m 49s		the patch passed
+1 💚	shadedclient	21m 15s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	620m 28s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	1m 13s		The patch does not generate ASF License warnings.
		720m 29s

Reason	Tests
Failed junit tests	hadoop.fs.contract.hdfs.TestHDFSContractRename
	hadoop.cli.TestAclCLIWithPosixAclInheritance
	hadoop.hdfs.server.namenode.TestEditLogRace
	hadoop.hdfs.server.namenode.TestDeleteRace
	hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
	hadoop.hdfs.TestSafeMode
	hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate
	hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
	hadoop.fs.TestWebHdfsFileContextMainOperations
	hadoop.fs.TestSWebHdfsFileContextMainOperations
	hadoop.security.TestPermission
	hadoop.hdfs.security.TestDelegationToken
	hadoop.hdfs.TestExtendedAcls
	hadoop.hdfs.TestFileLengthOnClusterRestart
	hadoop.hdfs.server.namenode.TestNameNodeAcl
	hadoop.hdfs.server.namenode.TestAuditLogs
	hadoop.hdfs.server.namenode.snapshot.TestSnapshotBlocksMap
	hadoop.hdfs.server.namenode.TestNameNodeRecovery
	hadoop.hdfs.server.namenode.TestGetBlockLocations
	hadoop.hdfs.server.namenode.TestDecommissioningStatusWithBackoffMonitor
	hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer
	hadoop.hdfs.TestDFSUpgradeFromImage
	hadoop.hdfs.TestAclsEndToEnd
	hadoop.hdfs.server.namenode.TestNamenodeRetryCache
	hadoop.fs.viewfs.TestViewFileSystemLinkRegex
	hadoop.hdfs.server.datanode.TestBatchIbr
	hadoop.fs.viewfs.TestViewFileSystemHdfs
	hadoop.fs.contract.hdfs.TestHDFSContractCreate
	hadoop.hdfs.TestFileAppend
	hadoop.hdfs.TestSafeModeWithStripedFile
	hadoop.hdfs.TestEncryptionZonesWithKMS
	hadoop.hdfs.server.namenode.snapshot.TestSnapshot
	hadoop.hdfs.server.namenode.snapshot.TestOrderedSnapshotDeletion
	hadoop.hdfs.server.namenode.ha.TestPendingCorruptDnMessages
	hadoop.hdfs.TestFileCreation
	hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForStoragePolicy
	hadoop.hdfs.TestErasureCodingPolicyWithSnapshot
	hadoop.hdfs.TestDFSRollback
	hadoop.hdfs.TestDFSClientFailover
	hadoop.fs.viewfs.TestViewFsHdfs
	hadoop.hdfs.server.datanode.TestBlockReplacement
	hadoop.hdfs.server.balancer.TestBalancer
	hadoop.hdfs.TestErasureCodingPolicyWithSnapshotWithRandomECPolicy
	hadoop.hdfs.TestFileAppendRestart
	hadoop.hdfs.TestDFSPermission
	hadoop.hdfs.TestFetchImage
	hadoop.hdfs.TestDFSUpgrade
	hadoop.hdfs.TestSetTimes
	hadoop.fs.TestSymlinkHdfsFileContext
	hadoop.hdfs.server.namenode.ha.TestBootstrapStandby
	hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits
	hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer
	hadoop.hdfs.web.TestWebHDFSAcl
	hadoop.hdfs.server.namenode.ha.TestUpdateBlockTailing
	hadoop.hdfs.server.namenode.TestNestedEncryptionZones
	hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
	hadoop.hdfs.server.namenode.TestFSEditLogLoader
	hadoop.hdfs.TestAppendSnapshotTruncate
	hadoop.hdfs.TestLocalDFS
	hadoop.fs.viewfs.TestViewFsLinkFallback
	hadoop.hdfs.TestDistributedFileSystemWithECFile
	hadoop.hdfs.TestErasureCodingExerciseAPIs
	hadoop.hdfs.server.namenode.TestReconstructStripedBlocks
	hadoop.hdfs.TestMaintenanceState
	hadoop.hdfs.TestQuota
	hadoop.hdfs.TestDFSShell
	hadoop.hdfs.TestRollingUpgrade
	hadoop.hdfs.TestListFilesInFileContext
	hadoop.fs.viewfs.TestViewFileSystemLinkMergeSlash
	hadoop.fs.viewfs.TestViewFileSystemLinkFallback
	hadoop.hdfs.server.namenode.TestFSImageWithAcl
	hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForAcl
	hadoop.hdfs.server.namenode.ha.TestStandbyIsHot
	hadoop.hdfs.server.namenode.TestFsck
	hadoop.hdfs.server.namenode.TestStorageRestore
	hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication
	hadoop.hdfs.server.namenode.TestAddStripedBlocks
	hadoop.fs.TestFcHdfsSetUMask
	hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS
	hadoop.hdfs.server.namenode.TestAddBlockRetry
	hadoop.hdfs.TestHDFSFileSystemContract
	hadoop.fs.TestGlobPaths
	hadoop.hdfs.TestLeaseRecovery2
	hadoop.hdfs.server.namenode.TestHDFSConcat
	hadoop.hdfs.web.TestFSMainOperationsWebHdfs
	hadoop.hdfs.TestBlockStoragePolicy
	hadoop.hdfs.TestViewDistributedFileSystemContract
	hadoop.hdfs.server.namenode.ha.TestStandbyBlockManagement
	hadoop.hdfs.TestDFSStartupVersions
	hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles
	hadoop.hdfs.TestEncryptionZonesWithHA
	hadoop.hdfs.server.namenode.snapshot.TestCheckpointsWithSnapshots
	hadoop.hdfs.server.blockmanagement.TestCorruptionWithFailover
	hadoop.hdfs.server.namenode.TestFileContextAcl
	hadoop.hdfs.server.namenode.TestAclConfigFlag
	hadoop.hdfs.TestDecommission
	hadoop.hdfs.TestRestartDFS
	hadoop.hdfs.server.namenode.TestNameEditsConfigs
	hadoop.hdfs.server.namenode.TestFSNamesystemLockReport
	hadoop.hdfs.TestDFSRename
	hadoop.hdfs.TestRollingUpgradeRollback
	hadoop.hdfs.TestMiniDFSCluster
	hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
	hadoop.hdfs.server.namenode.TestCacheDirectivesWithViewDFS
	hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
	hadoop.hdfs.server.blockmanagement.TestPendingDataNodeMessages
	hadoop.hdfs.server.namenode.TestFSImageWithSnapshot
	hadoop.hdfs.server.namenode.TestParallelImageWrite
	hadoop.hdfs.server.namenode.TestCommitBlockSynchronization
	hadoop.hdfs.server.namenode.TestPersistentStoragePolicySatisfier
	hadoop.fs.viewfs.TestViewFileSystemAtHdfsRoot
	hadoop.hdfs.server.namenode.TestXAttrConfigFlag
	hadoop.hdfs.server.namenode.snapshot.TestRandomOpsWithSnapshots
	hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
	hadoop.hdfs.web.TestWebHDFSXAttr
	hadoop.cli.TestAclCLI
	hadoop.hdfs.server.namenode.TestFSImage
	hadoop.hdfs.server.namenode.ha.TestHASafeMode
	hadoop.hdfs.server.namenode.TestAddBlock
	hadoop.hdfs.server.namenode.TestQuotaByStorageType
	hadoop.fs.TestFcHdfsCreateMkdir
	hadoop.hdfs.server.namenode.ha.TestQuotasWithHA
	hadoop.hdfs.server.balancer.TestBalancerService
	hadoop.fs.viewfs.TestViewFsAtHdfsRoot
	hadoop.hdfs.server.namenode.snapshot.TestSnapRootDescendantDiff
	hadoop.hdfs.TestRenameWhileOpen
	hadoop.hdfs.TestSafeModeWithStripedFileWithRandomECPolicy
	hadoop.hdfs.TestDistributedFileSystemWithECFileWithRandomECPolicy
	hadoop.fs.TestSymlinkHdfsFileSystem
	hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport
	hadoop.hdfs.TestReplication
	hadoop.hdfs.server.namenode.TestLargeDirectoryDelete
	hadoop.fs.contract.hdfs.TestHDFSContractAppend
	hadoop.hdfs.server.namenode.sps.TestStoragePolicySatisfierWithStripedFile
	hadoop.hdfs.server.namenode.snapshot.TestXAttrWithSnapshot
	hadoop.hdfs.TestTrashWithSecureEncryptionZones
	hadoop.hdfs.server.namenode.ha.TestHAMetrics
	hadoop.hdfs.TestTrashWithEncryptionZones
	hadoop.hdfs.server.namenode.snapshot.TestFSImageWithOrderedSnapshotDeletion
	hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
	hadoop.hdfs.server.namenode.ha.TestFailoverWithBlockTokensEnabled
	hadoop.hdfs.TestInjectionForSimulatedStorage
	hadoop.hdfs.TestFileCreationDelete
	hadoop.hdfs.server.namenode.TestListCorruptFileBlocks
	hadoop.fs.TestHDFSFileContextMainOperations
	hadoop.fs.viewfs.TestViewFileSystemOverloadSchemeHdfsFileSystemContract
	hadoop.fs.contract.hdfs.TestHDFSContractConcat
	hadoop.hdfs.server.namenode.ha.TestHAStateTransitions
	hadoop.cli.TestCryptoAdminCLI
	hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForContentSummary
	hadoop.hdfs.server.namenode.TestFileContextXAttr
	hadoop.hdfs.TestErasureCodingPoliciesWithRandomECPolicy
	hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
	hadoop.hdfs.TestDatanodeLayoutUpgrade
	hadoop.hdfs.TestDFSFinalize
	hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead
	hadoop.hdfs.server.namenode.TestFSNamesystemLock
	hadoop.hdfs.server.namenode.TestFSNamesystem
	hadoop.hdfs.TestDistributedFileSystem
	hadoop.hdfs.TestDFSStorageStateRecovery
	hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation
	hadoop.hdfs.server.namenode.TestNameNodeXAttr
	hadoop.hdfs.server.namenode.TestDecommissioningStatus
	hadoop.hdfs.server.namenode.TestCacheDirectives
	hadoop.hdfs.server.namenode.TestStartup
	hadoop.hdfs.TestRollingUpgradeDowngrade
	hadoop.hdfs.server.namenode.ha.TestHAAppend
	hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForXAttr
	hadoop.hdfs.server.namenode.snapshot.TestAclWithSnapshot
	hadoop.hdfs.TestEncryptionZones
	hadoop.hdfs.TestErasureCodingPolicies
	hadoop.hdfs.TestEncryptedTransfer
	hadoop.hdfs.server.namenode.TestSecondaryNameNodeUpgrade
	hadoop.hdfs.server.namenode.TestINodeFile
	hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
	hadoop.hdfs.TestAbandonBlock
	hadoop.hdfs.server.namenode.TestBackupNode
	hadoop.hdfs.web.TestWebHdfsFileSystemContract

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3205/4/artifact/out/Dockerfile
GITHUB PR	#3205
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux dcf1ebe5b6c8 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	fgl / `bf363db`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3205/4/testReport/
Max. process+thread count	3604 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3205/4/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

prasad-acit · 2021-07-22T19:44:13Z

Thanks @shvachko for review and feedback.

General suggestion here.

Seems there is a fair amount of refactoring going on in the patch, like adding createMissingDirs(). We can do such refactoring on trunk, and then this change will be smaller, which will make fgl branch maintenance simpler.
-- I got your point. I tried to accommodate these changes into Trunk, but the changes are more specific to FGL. Also, tried to avoid the refractor, but it lead to some duplicate code. We don't deviate much from the framework code. Try to keep the changes minimal.

It would be good to have a test one can run to verify the change works. Not necessarily a new test. Could be an existing one.
-- Yes, we do test with existing UTs. Basic functionalities - Create/Delete File/Dir are fine with FGL. There are other test failures, which we are working on.

…kaprasad C. (#3205)

…tory. Contributed by Renukaprasad C. (#3205)

github-actions · 2025-11-27T00:23:44Z

We're closing this stale PR because it has been open for 100 days with no activity. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you feel like this was a mistake, or you would like to continue working on it, please feel free to re-open it and ask for a committer to remove the stale tag and review again.
Thanks all for your contribution.

shvachko and others added 3 commits July 16, 2021 13:08

INodeMap with PartitionedGSet and per-partition locking.

0e7d216

Add namespace key for INode. (shv)

1829b58

HDFS-16125. [FGL] Fix the iterator for PartitionedGSet. Contributed b…

b1e2c07

…y Xing Lin. (apache#3197)

asfgit force-pushed the fgl branch from 9a60e53 to b1e2c07 Compare July 16, 2021 21:30

HDFS-16130: Implement CREATE File with FGL

f28db1d

prasad-acit force-pushed the HDFS-16130 branch from 9679236 to f28db1d Compare July 19, 2021 06:49

xinglin reviewed Jul 21, 2021

View reviewed changes

HDFS-16130. Create API implementation-Review comments

bf363db

prasad-acit force-pushed the HDFS-16130 branch from 6749094 to bf363db Compare July 21, 2021 14:14

shvachko reviewed Jul 21, 2021

View reviewed changes

HDFS-16130. Create API implementation-Review comments

507c76f

asfgit pushed a commit that referenced this pull request Jul 23, 2021

HDFS-16130. [FGL] Implement CREATE File with FGL. Contributed by Renu…

7598a68

…kaprasad C. (#3205)

asfgit force-pushed the fgl branch from b1e2c07 to 7598a68 Compare July 23, 2021 22:30

asfgit pushed a commit that referenced this pull request Jul 31, 2021

HDFS-16130. [FGL] Implement CREATE File with FGL. Contributed by Renu…

0f1a9a8

…kaprasad C. (#3205)

asfgit force-pushed the fgl branch from 7598a68 to 0f1a9a8 Compare July 31, 2021 20:06

asfgit pushed a commit that referenced this pull request Aug 13, 2021

HDFS-16130. [FGL] Implement CREATE File with FGL. Contributed by Renu…

88484fc

…kaprasad C. (#3205)

asfgit pushed a commit that referenced this pull request Aug 13, 2021

HDFS-16141. [FGL] Address permission related issues with File / Direc…

892fa48

…tory. Contributed by Renukaprasad C. (#3205)

asfgit force-pushed the fgl branch from 0f1a9a8 to 892fa48 Compare August 13, 2021 19:06

github-actions bot added the Stale label Nov 27, 2025

github-actions bot closed this Nov 28, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDFS-16130. Implement CREATE File with FGL #3205

HDFS-16130. Implement CREATE File with FGL #3205

prasad-acit commented Jul 14, 2021

Uh oh!

xinglin left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shvachko left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hadoop-yetus commented Jul 22, 2021

Uh oh!

prasad-acit commented Jul 22, 2021

Uh oh!

github-actions bot commented Nov 27, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

HDFS-16130. Implement CREATE File with FGL #3205

HDFS-16130. Implement CREATE File with FGL #3205

Conversation

prasad-acit commented Jul 14, 2021

Uh oh!

xinglin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shvachko left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hadoop-yetus commented Jul 22, 2021

Uh oh!

prasad-acit commented Jul 22, 2021

Uh oh!

github-actions bot commented Nov 27, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants