Filesystem_posix: GetFileList to include non-broken soft links. #7
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mooskagh
reviewed
Jun 2, 2018
DanielUranga
pushed a commit
to DanielUranga/lc0
that referenced
this pull request
Jun 16, 2019
cn4750
pushed a commit
to cn4750/lc0
that referenced
this pull request
Oct 19, 2020
merge lc0 master
Sopel97
pushed a commit
to Sopel97/lc0
that referenced
this pull request
Jul 4, 2021
Co-authored-by: borg323 <borg323@users.noreply.github.com>
Tilps
added a commit
that referenced
this pull request
Feb 26, 2024
* Add a mode to turn lc0 into a chunk data rescorer powered by Tablebase. * Add some stats. * Add secondary rescoring using wdl to reduce back propigation of blunders a bit. * Add policy distribution adjust support to rescorer. * Track the game outcomes, and the change to the start of the game * Add DTZ based assistance for secondary rescoring. * Change move count to a moves remaining to potentially use for modulating target value. * Use DTZ for pawnless 3 piece positions as a substitue for DTM to adjust move_count to be more correct * another fix. * More fixing. * Getting things compiling again. * Make rescorer more obvious. * reorder to match struct order. * Actually update the version when converting to v4 format. * Implement the threading support. * Fix compilation issues on some compilers. * More compilation fixing. * Fix off by one. * Add support for root probe policy boosting for minimum dtz in winning positions. * Fix test compile. * Fix missing option. * Add a counter. * Log if policy boost is for a move labelled illegal. * Add a histogram for total amount of boosted policy per boosted position. * Distribute boost rather than apply to all - also log before and after dists. * Add gaviotatb code for later use in dtm_boost * Fix compile issue on linux. * Prepare logic for dtm policy boost. * Load gaviota tb if specified. * Probe gaviota to decide which 'safe' moves are most deserving of boost based on dtm. * First attempt at supporting arbitrary starting point training data for rescorer. * Fix missing brackets. * Some fixes. * Avoid crashes from walking history before start of provided game information. * Some more merge fixes. * Fix some formatting. * Only process .gz files, don't crash out on invalid files, don't create output until input has been read. * Don't keep partially valid files. * Add basic range validation for input data. * Don't create writer any earlier than needed. * Fix decoding castling moves for the new Move format. * Validate game moves for legality. * Also log illegal move if it passes probability check but fails the real check. * Fix another merge error. * Compile fix for linux. * Plies left in rescorer (#1) * Rescore move_count using Gaviota TBs * Fix lczero-common commit * Add condition for Gaviota move_count rescoring * Post merge fixup for the kings/knights change in board. * Rescore tb v5 (#2) * Make lc0 output v5 training data. * Finish merge of v5 data into rescorer tb. * Fixes for rescoring v4 data. * Revert some unneeded formatting changes. * Support FRC input_format in rescoring. * Add some very important missing break statements... * Fix merge. * Change movement decode to not rely on there being any history planes filled in. Since that will not always be the case for input type 3. * Minimum changes to make it compile again post merge. * Input format 3 support. * Fix data range checks were incorrect for format 3 and 2. * Fix up bugs with chess 960 castle moves that leave a rook or king in place. * Post merge compile fixups for renames. * Add support for hectoplies and hectoplies armageddon to validate, and fixup the merge of latest code. * More fixes for type 4 and 132. * Add input format conversion support to rescorer. * Better match for training. * Add canonical v2 format to rescorer. * Add a utility for substituting policy from higher quality data into main data. * Fix missing option and add some commented out diagnostic code. * More cleanup in comments. * Handle empty policy-substitutions dir and input dir better. * Don't keep chunks that are marked as not for training. * More fixes for handling files with placeholder chunks. * Add 'deblunderer' Completely untested... * Fix some bugs in deblunder. * simplify windows rescorer build (#4) Co-authored-by: borg323 <borg323@users.noreply.github.com> * Tweak windows build file. * Some updates for writer.h/cc for v6 * Update rescorer loop.cc for V6. * Some additional validations to do with played_idx/best_idx. * make appveyor build the rescorer (#7) Co-authored-by: borg323 <borg323@users.noreply.github.com> * subproject for gaviota tb files (#8) Co-authored-by: borg323 <borg323@users.noreply.github.com> * 'Fix' for build on windows Probably should be fixed some other way... * Fix my breakage. (#9) * Update loop.cc * Update meson.build * Use the v6 field played_q to do a more direct blunder rescoring (#5) * included the issue 1308 deblunder mechanism in loop.cc * blunder detection now acts on missed proven wins and unforced proven losses * added comment on missing activeM * removed probabilistic randomization of result rescorer and worked with v6 data instead * included moves left rescore, removed unneeded options * doubled code not needed as final positions aren't special * changed appveyor script to hopefully build rescorer.sln * reverted failed attempt at fixing appveyor * included minimal std::cout for blunders * included blunder counter, added comment to visits v6 data checking * checking for bit 3 of invariance info to make sure best_q is a proven win * Fix v5 upgrading for decisive games. * Additional safety. * Add missing brackets. * don't keep the first TB position for the deblundering pass. (#10) * included the issue 1308 deblunder mechanism in loop.cc * blunder detection now acts on missed proven wins and unforced proven losses * added comment on missing activeM * removed probabilistic randomization of result rescorer and worked with v6 data instead * included moves left rescore, removed unneeded options * doubled code not needed as final positions aren't special * changed appveyor script to hopefully build rescorer.sln * reverted failed attempt at fixing appveyor * included minimal std::cout for blunders * included blunder counter, added comment to visits v6 data checking * checking for bit 3 of invariance info to make sure best_q is a proven win * don't keep the first TB position for rescorer * change recorer logo (#11) Co-authored-by: borg323 <borg323@users.noreply.github.com> * Make the deblunder transition soft through a width parameter (#13) * included the issue 1308 deblunder mechanism in loop.cc * blunder detection now acts on missed proven wins and unforced proven losses * added comment on missing activeM * removed probabilistic randomization of result rescorer and worked with v6 data instead * included moves left rescore, removed unneeded options * doubled code not needed as final positions aren't special * changed appveyor script to hopefully build rescorer.sln * reverted failed attempt at fixing appveyor * included minimal std::cout for blunders * included blunder counter, added comment to visits v6 data checking * checking for bit 3 of invariance info to make sure best_q is a proven win * don't keep the first TB position for rescorer * added a deblunder width parameter to allow a soft transition * clang formatting * resolve merge conflict * Add nnue plain file output (#12) * GetFen() from pr834 * first version of nnue output * flag to delete fils * address review comments * support pre v6 data * fix sign * correct nnue data misunderstanding Co-authored-by: borg323 <borg323@users.noreply.github.com> * fix copy-paste error (#15) Co-authored-by: borg323 <borg323@users.noreply.github.com> * add -t flag (#16) Co-authored-by: borg323 <borg323@users.noreply.github.com> * Post merge fixes. * Missed cleanup. * Fix input format change bug that can corrupt played_idx and best_idx * Post merge fixes. * fix merge * remove unnecessary options * split out rescore loop * minimize rescorer build * merge rescorer with master * minimize syzygy diff --------- Co-authored-by: Tilps <tilps@users.noreply.github.com> Co-authored-by: Henrik Forstén <henrik.forsten@gmail.com> Co-authored-by: borg323 <borg323@users.noreply.github.com> Co-authored-by: Naphthalin <40385638+Naphthalin@users.noreply.github.com>
PikaCat-OuO
pushed a commit
to official-pikafish/px0
that referenced
this pull request
Mar 20, 2024
* Add a mode to turn lc0 into a chunk data rescorer powered by Tablebase. * Add some stats. * Add secondary rescoring using wdl to reduce back propigation of blunders a bit. * Add policy distribution adjust support to rescorer. * Track the game outcomes, and the change to the start of the game * Add DTZ based assistance for secondary rescoring. * Change move count to a moves remaining to potentially use for modulating target value. * Use DTZ for pawnless 3 piece positions as a substitue for DTM to adjust move_count to be more correct * another fix. * More fixing. * Getting things compiling again. * Make rescorer more obvious. * reorder to match struct order. * Actually update the version when converting to v4 format. * Implement the threading support. * Fix compilation issues on some compilers. * More compilation fixing. * Fix off by one. * Add support for root probe policy boosting for minimum dtz in winning positions. * Fix test compile. * Fix missing option. * Add a counter. * Log if policy boost is for a move labelled illegal. * Add a histogram for total amount of boosted policy per boosted position. * Distribute boost rather than apply to all - also log before and after dists. * Add gaviotatb code for later use in dtm_boost * Fix compile issue on linux. * Prepare logic for dtm policy boost. * Load gaviota tb if specified. * Probe gaviota to decide which 'safe' moves are most deserving of boost based on dtm. * First attempt at supporting arbitrary starting point training data for rescorer. * Fix missing brackets. * Some fixes. * Avoid crashes from walking history before start of provided game information. * Some more merge fixes. * Fix some formatting. * Only process .gz files, don't crash out on invalid files, don't create output until input has been read. * Don't keep partially valid files. * Add basic range validation for input data. * Don't create writer any earlier than needed. * Fix decoding castling moves for the new Move format. * Validate game moves for legality. * Also log illegal move if it passes probability check but fails the real check. * Fix another merge error. * Compile fix for linux. * Plies left in rescorer (LeelaChessZero#1) * Rescore move_count using Gaviota TBs * Fix lczero-common commit * Add condition for Gaviota move_count rescoring * Post merge fixup for the kings/knights change in board. * Rescore tb v5 (LeelaChessZero#2) * Make lc0 output v5 training data. * Finish merge of v5 data into rescorer tb. * Fixes for rescoring v4 data. * Revert some unneeded formatting changes. * Support FRC input_format in rescoring. * Add some very important missing break statements... * Fix merge. * Change movement decode to not rely on there being any history planes filled in. Since that will not always be the case for input type 3. * Minimum changes to make it compile again post merge. * Input format 3 support. * Fix data range checks were incorrect for format 3 and 2. * Fix up bugs with chess 960 castle moves that leave a rook or king in place. * Post merge compile fixups for renames. * Add support for hectoplies and hectoplies armageddon to validate, and fixup the merge of latest code. * More fixes for type 4 and 132. * Add input format conversion support to rescorer. * Better match for training. * Add canonical v2 format to rescorer. * Add a utility for substituting policy from higher quality data into main data. * Fix missing option and add some commented out diagnostic code. * More cleanup in comments. * Handle empty policy-substitutions dir and input dir better. * Don't keep chunks that are marked as not for training. * More fixes for handling files with placeholder chunks. * Add 'deblunderer' Completely untested... * Fix some bugs in deblunder. * simplify windows rescorer build (LeelaChessZero#4) Co-authored-by: borg323 <borg323@users.noreply.github.com> * Tweak windows build file. * Some updates for writer.h/cc for v6 * Update rescorer loop.cc for V6. * Some additional validations to do with played_idx/best_idx. * make appveyor build the rescorer (LeelaChessZero#7) Co-authored-by: borg323 <borg323@users.noreply.github.com> * subproject for gaviota tb files (LeelaChessZero#8) Co-authored-by: borg323 <borg323@users.noreply.github.com> * 'Fix' for build on windows Probably should be fixed some other way... * Fix my breakage. (LeelaChessZero#9) * Update loop.cc * Update meson.build * Use the v6 field played_q to do a more direct blunder rescoring (LeelaChessZero#5) * included the issue 1308 deblunder mechanism in loop.cc * blunder detection now acts on missed proven wins and unforced proven losses * added comment on missing activeM * removed probabilistic randomization of result rescorer and worked with v6 data instead * included moves left rescore, removed unneeded options * doubled code not needed as final positions aren't special * changed appveyor script to hopefully build rescorer.sln * reverted failed attempt at fixing appveyor * included minimal std::cout for blunders * included blunder counter, added comment to visits v6 data checking * checking for bit 3 of invariance info to make sure best_q is a proven win * Fix v5 upgrading for decisive games. * Additional safety. * Add missing brackets. * don't keep the first TB position for the deblundering pass. (LeelaChessZero#10) * included the issue 1308 deblunder mechanism in loop.cc * blunder detection now acts on missed proven wins and unforced proven losses * added comment on missing activeM * removed probabilistic randomization of result rescorer and worked with v6 data instead * included moves left rescore, removed unneeded options * doubled code not needed as final positions aren't special * changed appveyor script to hopefully build rescorer.sln * reverted failed attempt at fixing appveyor * included minimal std::cout for blunders * included blunder counter, added comment to visits v6 data checking * checking for bit 3 of invariance info to make sure best_q is a proven win * don't keep the first TB position for rescorer * change recorer logo (LeelaChessZero#11) Co-authored-by: borg323 <borg323@users.noreply.github.com> * Make the deblunder transition soft through a width parameter (LeelaChessZero#13) * included the issue 1308 deblunder mechanism in loop.cc * blunder detection now acts on missed proven wins and unforced proven losses * added comment on missing activeM * removed probabilistic randomization of result rescorer and worked with v6 data instead * included moves left rescore, removed unneeded options * doubled code not needed as final positions aren't special * changed appveyor script to hopefully build rescorer.sln * reverted failed attempt at fixing appveyor * included minimal std::cout for blunders * included blunder counter, added comment to visits v6 data checking * checking for bit 3 of invariance info to make sure best_q is a proven win * don't keep the first TB position for rescorer * added a deblunder width parameter to allow a soft transition * clang formatting * resolve merge conflict * Add nnue plain file output (LeelaChessZero#12) * GetFen() from pr834 * first version of nnue output * flag to delete fils * address review comments * support pre v6 data * fix sign * correct nnue data misunderstanding Co-authored-by: borg323 <borg323@users.noreply.github.com> * fix copy-paste error (LeelaChessZero#15) Co-authored-by: borg323 <borg323@users.noreply.github.com> * add -t flag (LeelaChessZero#16) Co-authored-by: borg323 <borg323@users.noreply.github.com> * Post merge fixes. * Missed cleanup. * Fix input format change bug that can corrupt played_idx and best_idx * Post merge fixes. * fix merge * remove unnecessary options * split out rescore loop * minimize rescorer build * merge rescorer with master * minimize syzygy diff --------- Co-authored-by: Tilps <tilps@users.noreply.github.com> Co-authored-by: Henrik Forstén <henrik.forsten@gmail.com> Co-authored-by: borg323 <borg323@users.noreply.github.com> Co-authored-by: Naphthalin <40385638+Naphthalin@users.noreply.github.com> (cherry picked from commit 738c4aa)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This development makes GetFileList to include soft link when they actually resolve to a regular file.
The same simple development with just including DT_LNK is the allowed types could prevent lc0 to list the directory if a broken link was present in the directory.
Alternatively, we could add a "bool FileExists(cont char*)" in the filesystem interface and call the function.