Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Duplicate File List #1

Closed
gMan1990 opened this issue Jan 22, 2019 · 3 comments
Closed

Duplicate File List #1

gMan1990 opened this issue Jan 22, 2019 · 3 comments
Labels
bug Something isn't working good first issue Good for newcomers

Comments

@gMan1990
Copy link

gMan1990 commented Jan 22, 2019 •
edited

DupYSL.xlsx
count: 51

文件名称 路径 大小 文件类型 哈希值(sha-1) 组别
American Coots.mp3 \American Coots\ 16933108 .mp3 32d506acb2b9cade1a1d8c2168ed39fe44fa4ff3 1
American Coots.mp3 \Bird - American Coots\ 16933108 .mp3 32d506acb2b9cade1a1d8c2168ed39fe44fa4ff3 1
American Coots_NPS _ Neal Herbert_2016-09-14.jpg \American Coots\ 161684 .jpg 399afd7ba13dd3933a07e96dc437a2e7a43d8b44 2
American Coots_NPS _ Neal Herbert_2016-09-14.jpg \Bird - American Coots\ 161684 .jpg 399afd7ba13dd3933a07e96dc437a2e7a43d8b44 2
Sound Library - American Dipper.mp3 \American Dipper\ 1047833 .mp3 6f21b5047092caa9e9cb8f9524f40f4a4fc5ffa7 3
Sound Library - American Dipper.mp3 \Bird - American Dipper\ 1047833 .mp3 6f21b5047092caa9e9cb8f9524f40f4a4fc5ffa7 3
Sound Library - American Dipper_NPS _ Jacob W. Frank_2014-02-26.jpg \American Dipper\ 114918 .jpg ff0c2eb01f2a5ac30505622cc5669c9ffbdd4922 4
Sound Library - American Dipper_NPS _ Jacob W. Frank_2014-02-26.jpg \Bird - American Dipper\ 114918 .jpg ff0c2eb01f2a5ac30505622cc5669c9ffbdd4922 4
Sound Library - American Robin.mp3 \American Robin\ 598352 .mp3 ae10affc52a6ff0353f97bedced42b57dcf81e63 5
Sound Library - American Robin.mp3 \Bird - American Robin\ 598352 .mp3 ae10affc52a6ff0353f97bedced42b57dcf81e63 5
Sound Library - American Robin_NPS _ Neal Herbert_2016-05-06.jpg \American Robin\ 114606 .jpg 0401bf58b80dc125c5c062aa1274e7f821165c81 6
Sound Library - American Robin_NPS _ Neal Herbert_2016-05-06.jpg \Bird - American Robin\ 114606 .jpg 0401bf58b80dc125c5c062aa1274e7f821165c81 6
Sound Library - Anemone Geysers.mp3 \Anemone Geysers\ 9252144 .mp3 ccdd5b7af48afa90e170820e06721185e30e6e12 7
Sound Library - Anemone Geysers.mp3 \Geyser - Anemone\ 9252144 .mp3 ccdd5b7af48afa90e170820e06721185e30e6e12 7
Sound Library - Bald Eagle.mp3 \Bald Eagle\ 249314 .mp3 a11833187487dbc3e293da31d84b0defda6a5452 8
Sound Library - Bald Eagle.mp3 \Bird - Bald Eagle\ 249314 .mp3 a11833187487dbc3e293da31d84b0defda6a5452 8
Sound Library - Bald Eagle_NPS _ Jim Peaco_2016-09-14.jpg \Bald Eagle\ 44546 .jpg f831107feecec1f3495ab66f5b5c0c97c7278e8b 9
Sound Library - Bald Eagle_NPS _ Jim Peaco_2016-09-14.jpg \Bird - Bald Eagle\ 44546 .jpg f831107feecec1f3495ab66f5b5c0c97c7278e8b 9
Beehive Geyser.mp3 \Beehive Geyser\ 7438549 .mp3 a38eb8714ece33ac183b963c66d614df0a24e0cd 10
Beehive Geyser.mp3 \Geyser - Beehive\ 7438549 .mp3 a38eb8714ece33ac183b963c66d614df0a24e0cd 10
Canada Goose.mp3 \Bird - Canada Goose\ 1546384 .mp3 a4939a75dbf8d8912a3df7ec5b3ceb62372e777b 11
Canada Goose.mp3 \Canada Goose\ 1546384 .mp3 a4939a75dbf8d8912a3df7ec5b3ceb62372e777b 11
Common Loon.mp3 \Bird - Common Loon\ 3138301 .mp3 6070766a18f51d77b12fa220f7a690138e362a91 12
Common Loon.mp3 \Common Loon\ 3138301 .mp3 6070766a18f51d77b12fa220f7a690138e362a91 12
Sound Library - Common Raven.mp3 \Bird - Common Raven\ 410625 .mp3 5aba13dc8d0efcddf4c1514e5d05b9532874ea5b 13
Sound Library - Common Raven.mp3 \Common Raven\ 410625 .mp3 5aba13dc8d0efcddf4c1514e5d05b9532874ea5b 13
Sound Library - Common Raven_NPS_Neal Herbert_2013-10-23.jpg \Bird - Common Raven\ 21668 .jpg eee926813d906694df1a084a68463dff493fcf7a 14
Sound Library - Common Raven_NPS_Neal Herbert_2013-10-23.jpg \Common Raven\ 21668 .jpg eee926813d906694df1a084a68463dff493fcf7a 14
Sound Library - Killdeer.mp3 \Bird - Killdeer\ 2404224 .mp3 c804a184d1060941b5cb9c82e15bc115b307a982 15
Sound Library - Killdeer.mp3 \Killdeer\ 2404224 .mp3 c804a184d1060941b5cb9c82e15bc115b307a982 15
Sound Library - Mountain Bluebird.mp3 \Bird - Mountain Bluebird\ 419569 .mp3 5fa26cab7f22050b25da62e89bfe5bd510e666b1 16
Sound Library - Mountain Bluebird.mp3 \Mountain Bluebird\ 419569 .mp3 5fa26cab7f22050b25da62e89bfe5bd510e666b1 16
Sound Library - Mountain Bluebird_NPS _ Neal Herbert_2015-06-17.jpg \Bird - Mountain Bluebird\ 57960 .jpg c2c7e958e71ac92399c4c6b17b87981faf7d2791 17
Sound Library - Mountain Bluebird_NPS _ Neal Herbert_2015-06-17.jpg \Mountain Bluebird\ 57960 .jpg c2c7e958e71ac92399c4c6b17b87981faf7d2791 17
Sound Library - Ruffed Grouse.mp3 \Bird - Ruffed Grouse\ 430257 .mp3 bdcb7979c0b6158ba52757e214f342ffc4a56f5a 18
Sound Library - Ruffed Grouse.mp3 \Ruffed Grouse\ 430257 .mp3 bdcb7979c0b6158ba52757e214f342ffc4a56f5a 18
Sound Library - Ruffed Grouse_NPS_Neal Herbert_2014-05-25.jpg \Bird - Ruffed Grouse\ 46386 .jpg df23000bab4f385cee87102c2fe464abd1d6f818 19
Sound Library - Ruffed Grouse_NPS_Neal Herbert_2014-05-25.jpg \Ruffed Grouse\ 46386 .jpg df23000bab4f385cee87102c2fe464abd1d6f818 19
Sound Library - Sandhill Crane.mp3 \Bird - Sandhill Crane\ 607404 .mp3 05dd93a84bdb5be2423606ca06db94deeac0bd14 20
Sound Library - Sandhill Crane.mp3 \Crane (sandhill)\ 607404 .mp3 05dd93a84bdb5be2423606ca06db94deeac0bd14 20
Sound Library - Sandhill Crane.mp3 \Sandhill Crane\ 607404 .mp3 05dd93a84bdb5be2423606ca06db94deeac0bd14 20
Sound Library - Sandhill Crane_NPS_Neal Herbert_2015-04-25.jpg \Bird - Sandhill Crane\ 141661 .jpg 8f75aa6ad0ba1190564a757cd736eb6a505c3b09 21
Sound Library - Sandhill Crane_NPS_Neal Herbert_2015-04-25.jpg \Crane (sandhill)\ 141661 .jpg 8f75aa6ad0ba1190564a757cd736eb6a505c3b09 21
Sound Library - Sandhill Crane_NPS_Neal Herbert_2015-04-25.jpg \Sandhill Crane\ 141661 .jpg 8f75aa6ad0ba1190564a757cd736eb6a505c3b09 21
Townsend's Solitaire.mp3 \Bird - Townsend's Solitaire\ 264272 .mp3 a5a59394955b688bf1d9fb8430b12c7a668cb899 22
Townsend's Solitaire.mp3 \Townsend's Solitaire\ 264272 .mp3 a5a59394955b688bf1d9fb8430b12c7a668cb899 22
Warbling Vireos.mp3 \Bird - Warbling Vireo\ 1086306 .mp3 9fa4f6d5f83d44037ae8240e85e223fa1b3dd291 23
Warbling Vireos.mp3 \Warbling Vireo\ 1086306 .mp3 9fa4f6d5f83d44037ae8240e85e223fa1b3dd291 23
Warbling Vireos_NPS _ Neal Herbert_2015-07-05.jpg \Bird - Warbling Vireo\ 64861 .jpg f462a87f91c9be3b8c0aa4370af56b939085057c 24
Warbling Vireos_NPS _ Neal Herbert_2015-07-05.jpg \Warbling Vireo\ 64861 .jpg f462a87f91c9be3b8c0aa4370af56b939085057c 24
Western Meadow Lark.mp3 \Bird - Western Meadow Lark\ 586625 .mp3 72802395e6aeb1f003570286b25f4d72e91fb954 25
Western Meadow Lark.mp3 \Western Meadow Lark\ 586625 .mp3 72802395e6aeb1f003570286b25f4d72e91fb954 25
Sound Library - Wilson's Snipe.mp3 \Bird - Wilson's Snipe\ 6225714 .mp3 117348e4c60346516612e25b6869f3cbcfc408fe 26
Sound Library - Wilson's Snipe.mp3 \Wilson's Snipe\ 6225714 .mp3 117348e4c60346516612e25b6869f3cbcfc408fe 26
Sound Library - Wilson's Snipe_NPS _ Neal Herbert_2014-05-09.jpg \Bird - Wilson's Snipe\ 95106 .jpg 246f43f17bc54a3ac32ab876ec8f1a9b1fd4dc6a 27
Sound Library - Wilson's Snipe_NPS _ Neal Herbert_2014-05-09.jpg \Wilson's Snipe\ 95106 .jpg 246f43f17bc54a3ac32ab876ec8f1a9b1fd4dc6a 27
Sound Library - Dawn Chorus.mp3 \Birds - Dawn Chorus\ 1501792 .mp3 3f56724be7ac25f3287e293c90023c75aab4aa95 28
Sound Library - Dawn Chorus.mp3 \Dawn Chorus\ 1501792 .mp3 3f56724be7ac25f3287e293c90023c75aab4aa95 28
Sound Library - Dawn Chorus_NPS_Neal Herbert_2014-06-28.jpg \Birds - Dawn Chorus\ 39626 .jpg 2b76365e9ae83ca1bf5f3088fe6599f0b659f5eb 29
Sound Library - Dawn Chorus_NPS_Neal Herbert_2014-06-28.jpg \Dawn Chorus\ 39626 .jpg 2b76365e9ae83ca1bf5f3088fe6599f0b659f5eb 29
Sound Library - Chorus Frogs.mp3 \Boreal Chorus Frogs\ 2724030 .mp3 014a06f7c691e2c58cf625eed6fc74a4d46027de 30
Sound Library - Chorus Frogs.mp3 \Frogs (boreal chorus)\ 2724030 .mp3 014a06f7c691e2c58cf625eed6fc74a4d46027de 30
Sound Library - Chorus Frogs_NPS _ Neal Herbert_2015-04-27.jpg \Boreal Chorus Frogs\ 67052 .jpg dc3c68c23b15ffa565a560028692d737e94bd2aa 31
Sound Library - Chorus Frogs_NPS _ Neal Herbert_2015-04-27.jpg \Frogs (boreal chorus)\ 67052 .jpg dc3c68c23b15ffa565a560028692d737e94bd2aa 31
Sound Library - Castle Geyser.mp3 \Castle Geyser\ 3970457 .mp3 6d5e950158ad0683d0c45635088ab640479aadd1 32
Sound Library - Castle Geyser.mp3 \Geyser - Castle\ 3970457 .mp3 6d5e950158ad0683d0c45635088ab640479aadd1 32
Sound Library - Castle Geyser_NPS_Neal Herbert_2014-02-26.jpg \Castle Geyser\ 33659 .jpg 4a5aed93c562445a8fe5a5c36670c4a8ebc4ba32 33
Sound Library - Castle Geyser_NPS_Neal Herbert_2014-02-26.jpg \Geyser - Castle\ 33659 .jpg 4a5aed93c562445a8fe5a5c36670c4a8ebc4ba32 33
Sound Library - Cliff Geyser.mp3 \Cliff Geyser\ 648700 .mp3 0714b7b229140719f95fe5f9102dc3d9f39684aa 34
Sound Library - Cliff Geyser.mp3 \Geyser - Cliff\ 648700 .mp3 0714b7b229140719f95fe5f9102dc3d9f39684aa 34
Sound Library - Cliff Geyser_NPS_Neal Herbert_2013-10-23.jpg \Cliff Geyser\ 35533 .jpg e374fe2432f17aa3f0c10cd5fe880d3ff4cbb6eb 35
Sound Library - Cliff Geyser_NPS_Neal Herbert_2013-10-23.jpg \Geyser - Cliff\ 35533 .jpg e374fe2432f17aa3f0c10cd5fe880d3ff4cbb6eb 35
Sound Library - The Dragon's Mouth.mp3 \Dragon's Mouth Spring\ 2663357 .mp3 226253c954303ec925a21c8dbab322e81c87118c 36
Sound Library - The Dragon's Mouth.mp3 \The Dragon's Mouth\ 2663357 .mp3 226253c954303ec925a21c8dbab322e81c87118c 36
Sound Library - The Dragon's Mouth_NPS_Neal Herbert_2013-10-29.jpg \Dragon's Mouth Spring\ 49273 .jpg 18a981e5f30f39ca2ee7167900d23a8b114ffa30 37
Sound Library - The Dragon's Mouth_NPS_Neal Herbert_2013-10-29.jpg \The Dragon's Mouth\ 49273 .jpg 18a981e5f30f39ca2ee7167900d23a8b114ffa30 37
Sound Library - Fire.mp3 \Fire\ 1458947 .mp3 2ccafd9b50fffb305227ebfdb81c2e007af08b69 38
Sound Library - Fire.mp3 \Wildfire\ 1458947 .mp3 2ccafd9b50fffb305227ebfdb81c2e007af08b69 38
Sound Library - Fire_NPS_Jennifer Jerrett_2016-08-14.jpg \Fire\ 184265 .jpg 8b9ebcc38702363410bc1c7397d13adfdbf29265 39
Sound Library - Fire_NPS_Jennifer Jerrett_2016-08-14.jpg \Wildfire\ 184265 .jpg 8b9ebcc38702363410bc1c7397d13adfdbf29265 39
Sound Library - Fountain Paint Pot.mp3 \Fountain Paint Pot\ 1542693 .mp3 da43c00e6d0bdc2aec2f63b9a07d305a3fb7f030 40
Sound Library - Fountain Paint Pot.mp3 \Mud Pots\ 1542693 .mp3 da43c00e6d0bdc2aec2f63b9a07d305a3fb7f030 40
Sound Library - Fountain Paint Pot_NPS_Neal Herbert_2014-05-01.jpg \Fountain Paint Pot\ 252034 .jpg a92c716e12803a109fc51f1e10b9a1e792fc29b7 41
Sound Library - Fountain Paint Pot_NPS_Neal Herbert_2014-05-01.jpg \Mud Pots\ 252034 .jpg a92c716e12803a109fc51f1e10b9a1e792fc29b7 41
Sound Library - Grand Geyser.mp3 \Geyser - Grand\ 17098400 .mp3 cdd5b57f8b05cf6a68eb0bd86fc9a954cf8c9c91 42
Sound Library - Grand Geyser.mp3 \Grand Geyser\ 17098400 .mp3 cdd5b57f8b05cf6a68eb0bd86fc9a954cf8c9c91 42
Sound Library - Old Faithful.mp3 \Geyser - Old Faithful\ 9363751 .mp3 8e8a862274db012c3eab05cf78164b0feafa3b1e 43
Sound Library - Old Faithful.mp3 \Old Faithful Geyser\ 9363751 .mp3 8e8a862274db012c3eab05cf78164b0feafa3b1e 43
Sound Library - Old Faithful_NPS_Neal Herbert_2014-11-16.jpg \Geyser - Old Faithful\ 35675 .jpg d811e07be4152b800964478f6fd82896c1c19ba6 44
Sound Library - Old Faithful_NPS_Neal Herbert_2014-11-16.jpg \Old Faithful Geyser\ 35675 .jpg d811e07be4152b800964478f6fd82896c1c19ba6 44
Sound Library - Sawmill Geyser.mp3 \Geyser - Sawmill\ 1519118 .mp3 0888a6905578bf24a9dc5279b3a0dff636f07788 45
Sound Library - Sawmill Geyser.mp3 \Sawmill Geyser\ 1519118 .mp3 0888a6905578bf24a9dc5279b3a0dff636f07788 45
Sound Library - Sawmill Geyser_NPS_Neal Herbert_2014-02-26.jpg \Geyser - Sawmill\ 30742 .jpg 316d226a4a1f3284fb586af94e7c0ef846409700 46
Sound Library - Sawmill Geyser_NPS_Neal Herbert_2014-02-26.jpg \Sawmill Geyser\ 30742 .jpg 316d226a4a1f3284fb586af94e7c0ef846409700 46
Sound Library - Veteran Geyser.mp3 \Geyser - Veteran\ 3844134 .mp3 0f580fdc772263508de00f5ff76e5d7fd591a6c4 47
Sound Library - Veteran Geyser.mp3 \Veteran Geyser\ 3844134 .mp3 0f580fdc772263508de00f5ff76e5d7fd591a6c4 47
Sound Library - Vixen Geyser.mp3 \Geyser - Vixen\ 5692414 .mp3 8a23267f45cd59bcff82a7983ff44405db0142b6 48
Sound Library - Vixen Geyser.mp3 \Vixen Geyser\ 5692414 .mp3 8a23267f45cd59bcff82a7983ff44405db0142b6 48
Sound Library - Vixen Geyser_NPS_Neal Herbert_2015-03-13.jpg \Geyser - Vixen\ 223160 .jpg 49dde3147934902a941796fb64af26bbf1072734 49
Sound Library - Vixen Geyser_NPS_Neal Herbert_2015-03-13.jpg \Vixen Geyser\ 223160 .jpg 49dde3147934902a941796fb64af26bbf1072734 49
Sound Library - Singing Lake.mp3 \Singing Lake\ 3265559 .mp3 ba17376337e0c9001283534571b0ff418a4687aa 50
Sound Library - Singing Lake.mp3 \Yellowstone Lake (singing)\ 3265559 .mp3 ba17376337e0c9001283534571b0ff418a4687aa 50
Spadefoot Toad.mp3 \Spadefoot Toad\ 1447975 .mp3 eaac6b9b9ec2fcc7dc300ff040855077b8c32da4 51
Spadefoot Toad.mp3 \Toad (Spadefoot)\ 1447975 .mp3 eaac6b9b9ec2fcc7dc300ff040855077b8c32da4 51
@rosuH rosuH added bug Something isn't working good first issue Good for newcomers labels Jan 22, 2019
@rosuH
Copy link
Owner

rosuH commented Jan 22, 2019

@gMan1990 Thank you! I had removed all duplicated files in 20dbbcf.

@rosuH rosuH closed this as completed Jan 22, 2019
@gMan1990
Copy link
Author

gMan1990 commented Jan 22, 2019 •
edited

@rosuH 貌似md5验证文件是否相同是不严谨的:
http://bbs.eyuyan.com/read.php?tid=221786
http://bbs.eyuyan.com/read.php?tid=221786&displayMode=1&page=2#1359827

@rosuH
Copy link
Owner

rosuH commented Jan 22, 2019

@gMan1990 感谢回复~

MD5 的风险在于，任何第三方可以通过给当前文件附加乱序字节前缀而修改当前文件的 MD5 值。最终达到与目标文件相同的 MD5 值以便发起攻击。

但是在此项目的应用中：

  • 无第三方获得文件，也没有主动发起的碰撞攻击
    • 排除中间人情况，因为黄石公园的网页也没有放出 MD5 值或其他校验值，所以本身来源就是不可靠的
  • MD5 自然产生的碰撞
    • 概率极低，应该不太可能出现在 100 多个文件中...

当然，以上仅是基于我个人浅薄的知识储备，可能并不准确，欢迎探讨哈~😄

如果需要，我也可以写一个 SHA1 或 SHA256 的版本😂

参见：

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working good first issue Good for newcomers
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@gMan1990 @rosuH