Skip to content
This repository has been archived by the owner on Dec 16, 2021. It is now read-only.

modify DirectoryCopier #29

Closed
wants to merge 2 commits into from
Closed

modify DirectoryCopier #29

wants to merge 2 commits into from

Conversation

hn5092
Copy link

@hn5092 hn5092 commented Oct 11, 2016

in the Stage2KerberosDirectoryCopyMapper ,we are skip hiddenFile,so,when we begin stage3 , we first compare src adn dst file number size by every dir , if this equalDirs use empty hiddenfilefilter, it must be false, so add hiddenFileFilter

in the Stage2KerberosDirectoryCopyMapper ,we are skip hiddenFile,so,when we begin stage3 , we first compare src adn dst file number size by every dir , if this equalDirs use empty hiddenfilefilter,  it must be false, so add  hiddenFileFilter
@hn5092
Copy link
Author

hn5092 commented Oct 11, 2016

here is our log!
INFO [main] com.airbnb.reair.common.FsUtils: Number of files in hdfs://pajkcluster/user/hive/warehouse/jkgj_log.db/track_raw_wap211_log/pt=2016010600 (9) and hdfs://bxpajkcluster/user/hive/warehouse/jkgj_log.db/track_raw_wap211_log/pt=2016010600 (8) do not match!

@plypaul
Copy link
Collaborator

plypaul commented Oct 11, 2016

The DirectoryCopier is used by the incremental replication process too, so the filter may be better passed an argument to the function call.

@hn5092
Copy link
Author

hn5092 commented Oct 13, 2016

OK.i recommit for a later

@hn5092 hn5092 closed this Nov 12, 2016
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants