Fix TL-NET logging - #2628
Merged
Merged
Conversation
The Joint stage wrote relative AE and T-flank errors under the shared Train_Err/Val_Err header, so its columns did not mean what the header said and carried no absolute errors at all. Each stage now has its own AE, T-flank, and joint columns and leaves the ones it does not compute blank.
The scheduler was only stepped in the joint loop, so decay_lr did nothing during the AE and T-flank stages and the logged LR was always the base rate. Each stage now restarts the schedule at the base rate and steps it every epoch, which leaves the joint stage's learning rate identical to before while the earlier two stages finally anneal. Sharing one continuous schedule across the stages would instead have made the joint rate depend on the AE and T-flank epoch counts.
akenmorris
added a commit
that referenced
this pull request
Aug 12, 2026
* Derive the docs version from CMakeLists.txt The docs deploy hardcoded version 6.7 with the title "6.7 (dev)" and the "dev" alias, so every master build kept republishing into 6.7 even after 6.7 shipped and master moved on to 6.8.0-dev. Read the major/minor version out of CMakeLists.txt instead, so bumping the version is all that is needed to start a new dev version in the dropdown. Also let the docs deploy build from a ref other than master, so a released version's docs can be rebuilt from its tag and given the "latest" alias. Run the Mac Arm64 Build workflow manually with the tag selected, deploy_docs checked and docs_alias set to latest. * Switch RunUseCase.py dataset downloads to web download (#2627) * Download use case data over HTTPS instead of the cloud portal The use cases fetched their data through swcc from ShapeWorks Cloud, which has been dropping content: the femur_cut project came back with 49 meshes in a flat layout and no constraints at all, so the glob still matched three stale tiny_test meshes and the use case silently ran on 3 shapes. Datasets are now published as one zip per dataset alongside a manifest, fetched with plain urllib. Each archive is verified against its SHA-256 and file count before it is unpacked, and the marker records the checksum installed so republishing a dataset re-downloads it rather than reusing whatever is on disk. Nothing needs an account, and portal.py is untouched and still available for cloud work. Also fixes the same silent-failure bug in both ellipsoid_multiple_domain use cases, whose full branch globbed a path missing the inner directory level the tiny_test branch uses. Support/build_dataset_archives.py builds the archives and Support/verify_dataset_archives.py checks every archive against the paths the use cases actually glob for. * Version dataset archives individually instead of by directory Re-versioning every dataset because one changed is what made use-case-data-v1 through v3 expensive. Archives are now published as <name>-v<version>.zip and are never overwritten, with the manifest deciding which file a dataset name resolves to. Changing one dataset means bumping its version in DATASETS, rebuilding just that one, and uploading a single zip plus the regenerated manifest; everything else keeps the file it already had and the previous version stays on the server to roll back to. A full rebuild of unchanged data is now a no-op rather than 5 GB of work. Also stops verify_dataset_archives.py reporting scratch scripts: it now only checks use cases RunUseCase.py can actually invoke, which drops the false positive on ellipsoid_fd_cut.py, a variant that builds the particles directory it globs at runtime by copying from a previous ellipsoid_cut run. * Pin dataset versions in the source tree, not a server manifest Resolving dataset names through a manifest fetched at runtime meant an old release would pick up whatever was published later, which is exactly the isolation the use-case-data-vN directories existed to provide. Which archive a dataset resolves to is now pinned in datasets.json next to the module and checked into the source tree, so a release downloads the data it shipped against and republishing for a newer release cannot change what it gets. Updating a dataset means building a new version, uploading it, and pointing datasets.json at it; rolling back is a revert of that file. This also makes verification worth doing. Reading the checksums from the same server that serves the data protected against a truncated transfer but not against a bad re-upload, which is the failure that started this. Resolving locally removes the manifest round trip too, so re-running a use case on data already present needs no network at all. * Add a post-upload check that the server matches datasets.json One HEAD request per dataset compares the size the server reports against the pinned size, so a missing or half-uploaded archive is caught right after uploading rather than by a user running a use case. --full downloads each one and verifies its checksum. * Download tutorial data in the notebooks instead of via the Girder UI The four getting-started notebooks told users to register on the data portal, log in, and click through Girder to download a dataset by hand, which stops working once Girder is shut down. The same archives are already served as plain static files, so the notebooks now fetch and extract what they need in a cell, skipping the download if the folder is already present. No account is required and the obsolete portal walkthrough video is gone. The datasets are unchanged: ellipsoid_1mode for the segmentation and mesh notebooks, femur-v0 for data augmentation, both from use-case-data-v2. Neither is folded into the pinned use case index since femur-v0 is 4.3 GB and would have to be rebuilt and re-uploaded to gain nothing. * Fix VerifyUseCases.py for swpython * Fix TL-NET logging (#2628) * Fix TL-net train_log.csv column alignment (#2618) The Joint stage wrote relative AE and T-flank errors under the shared Train_Err/Val_Err header, so its columns did not mean what the header said and carried no absolute errors at all. Each stage now has its own AE, T-flank, and joint columns and leaves the ones it does not compute blank. * Apply decay_lr to the TL-net AE and T-flank stages (#2618) The scheduler was only stepped in the joint loop, so decay_lr did nothing during the AE and T-flank stages and the logged LR was always the base rate. Each stage now restarts the schedule at the base rate and steps it every epoch, which leaves the joint stage's learning rate identical to before while the earlier two stages finally anneal. Sharing one continuous schedule across the stages would instead have made the joint rate depend on the AE and T-flank epoch counts. * Derive the docs version from CMakeLists.txt The docs deploy hardcoded version 6.7 with the title "6.7 (dev)" and the "dev" alias, so every master build kept republishing into 6.7 even after 6.7 shipped and master moved on to 6.8.0-dev. Read the major/minor version out of CMakeLists.txt instead, so bumping the version is all that is needed to start a new dev version in the dropdown. Also let the docs deploy build from a ref other than master, so a released version's docs can be rebuilt from its tag and given the "latest" alias. Run the Mac Arm64 Build workflow manually with the tag selected, deploy_docs checked and docs_alias set to latest.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolve #2618 by adjusting the logging, also fix the scheduler stepping.