Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Deepspeed Feature Branch - merge master (#6193)
* docs: Improvements to HPC launcher docs (#6042) * Provide inline info about agent-specific scheduling options that do not apply to HPC Launcher configurations. * Identify enroot-specific differences from docker (like for Singularity) * Provide reference to custom resource pools as an option to deal with non-homogenous Slurm/PBS partitions. * chore: Allow newer Node versions 17-19 (#6038) * fix: k8s rm gives wrong slot count in rendezvous (#6044) * chore: bump version: 0.19.12-dev0 -> 0.20.0-dev0 (#6048) * chore: remove `applicableRoutespace` (#6040) * chore: warning fixes in web (#6041) * chore: fix warnings * chore: change eslint rules * chore: fix gpu nightly errors (#6046) * chore: missing nodev18 (#6050) * chore: add a dedicated exception for cli errors (#5649) switch sys.exit calls in cli with a new user-facing exception. * fix: SSO layout (#6053) * chore: clean up UI kit (#6039) * fix: lopsided training with 2,1 gpus (#6054) There was a guard to skip local zmq setup when local_size < 2, but that became no longer valid when local_size varied from worker to worker. The result is one extra global allgather in some cases, no big deal. * docs: add rbac ntsc & mr release notes (#6049) * chore: manual bump version (#6058) * ci: retry downloading GKE auth plugin [DET-8956] (#6056) We got a failure due to a timeout on this, so let it retry a few times. * docs: update Singularity known issues. (#6047) * fix: Add #rank to worker segment instead of timestamp segment of Pytorch Profiler files [MLG-326] (#6037) * Add pytorch profiler specific handling logic for appending rank to file name * Change to use f-string * fix only file name being passed in * remove print statement * fix: handle agent shutdown msg (#6065) * chore: manually bump vite version (#6066) * feat: Display better x-axis ticks on charts with time axis [WEB-849] (#6051) * remove xTickValues from props now that it can be calculated internally * test: add logging to a flaky test (#6068) Test is flaky but hard to pin down, so add some prints for next time. * fix: Unrelated models are shown in a workspace model registry tab (#6067) * feat: Added task-based historical allocation endpoint [DET-8537] (#6015) * fix: show `not found` and `spinner` properly (#6070) * fix: show `not found` and `spinner` properly * chore: change home redirect path * fix: projectDetail page * fix: project.workspaceId * build(deps): bump golang.org/x/text from 0.3.5 to 0.3.8 in /proto (#6061) * build(deps): bump golang.org/x/text from 0.3.5 to 0.3.8 in /proto Bumps [golang.org/x/text](https://github.com/golang/text) from 0.3.5 to 0.3.8. - [Release notes](https://github.com/golang/text/releases) - [Commits](golang/text@v0.3.5...v0.3.8) --- updated-dependencies: - dependency-name: golang.org/x/text dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * fix: gpt-neox docker image and startup hook to work for non-provileged user (#6060) * test: add locking around migration [DET-8957] (#6071) In integration tests, multiple processes can attempt to run migrations against the same database at once, which can lead to errors because PostgreSQL's `CREATE TABLE IF NOT EXISTS` is not great with concurrency (it allows for a time-of-check/time-of-use failure). The specific errors we were seeing were conflicts in the pg_type table, so the code now locks that table for the duration of the migration transaction. More information: https://www.postgresql.org/message-id/CA+TgmoZAdYVtwBfp1FL2sMZbiHCWT4UPrzRLNnX1Nb30Ku3-gg@mail.gmail.com https://stackoverflow.com/questions/29900845 * fix: logging inconsistent newlines in slurm (#6074) * fix: checkpoint helper for points > 1000 and points > maxDatapoints (#6069) * fix: replace migration table lock with advisory lock (#6077) Taking a table lock sometimes runs into permissions issues; advisory locking should avoid that. Also, I realized the locking should probably be after the deferred transaction close instead of before. * build: check npm version on install (#6079) This removes the `check-requirements` make target in the react folder and replaces it with npm's native version check against the engines property. This should make managing the node version slightly easier because there's one less place to check. * build: Apply webui lint fixes in precommit (#6078) * allow linters to fix in precommit This updates the web linters to automatically apply fixes when doing a pre-commit check. This should ideally streamline the commit process to reduce the amount of times the user needs to run prettier and eslint before committing. * tweak stylelint and eslint commands * stage changed files * type file_paths argument * ci: adjust target accuracy for a test (#6085) We got one failure [1] where the accuracy ended up just a hair below 0.83, so drop the target. [1] https://app.circleci.com/pipelines/github/determined-ai/determined/33883/workflows/4a5d3257-6061-4f4d-bd66-096a580a5959/jobs/1194282/steps * chore: UserBadge moves into design kit (#6086) * chore: Remove chart feature flag, remove unused code [WEB-930] (#6064) * tooltipsPlugin and TrialDetailsOverview alternates go into place * checkpoint helper for points > 1000 and points > maxDatapoints * move former LearningCurveChart into TrialsComparison * sync up with #6069 changes * fix: replace defaultvalue with initialValue (#6076) * fix: Dont suggest moving model into its current workspace (#6088) * ci: delete database at beginning of det deploy tests [DET-8937] (#6089) Previously, the database was being retained between tests, sometimes causing tests to fail when extra agents appeared due to agent reattach. The tests should generally be independent anyway, so reset the database (by default, with an option to disable) each time the cluster or master comes up. * feat: add Facepile component (#6081) * fix: pre-commit web bug fix (#6090) * ci: make GKE test jobs run serially (#6096) We keep hitting GKE GPU quotas; this will probably help with that. * fix: GPU counting for k8s cluster info page (#6094) * fix: test-e2e-gke-parallel use t4 (#6093) * ci: retry protoc download (#6095) We got an incorrect file downloaded one time [1], so retry this download, like in 2906257 (#5996). [1] https://app.circleci.com/pipelines/github/determined-ai/determined/34074/workflows/e48681f8-8b75-4349-82eb-06e922d8bfcb/jobs/1202610 * refactor: add Card to UI Kit [WEB-818] (#5893) * docs: Launcher doc improvements (#6099) - Generalize journalctl command example --since option to work on Unbutu. - Clarify user_name/group_name account requirements. * feat: Attend to TODOS accross the code base (#6087) * perf: tweak metrics series query. (#6105) * chore: race could cause run container to return a different error than expected [DET-8870] (#6092) * chore: add more metadata to slurm logs (#6030) * chore: remove `ExpCompareMetricNames`, `ExpCompareTrialsSample` endpoints. (#6106) * docs: fix reported DataPoint label doc (#6107) * fix: tolerate additional non-CPU, non-GPU quotas in k8s (#6109) * fix: stop filtering of valid options to reflect build issues (#6116) * fix: modal theme color (#6117) * fix: add bgColor in trial comparison table (#6119) * fix: browser console warnings (#6122) * fix: browser console warnings * fix: remove spread operator * chore: UIKit Pivot renaming (#6120) * fix: correct minor JSX syntax (#6126) * docs: add myst_parser extension (#6127) We would like to support markdown-format documentation. There are still some kinks to be worked out with converting rst to myst files, but this is a start. * docs: fix some broken redirects (#6129) * feat: add 3rd batch of TODO removals (#6115) * feat: generic proxy configs [DET-8761] (#5978) * build: [MLG-336] Limit the version of protobuf (#6134) build: [MLG-336] Limit the version of protobuf Installing the requirement `tensorflow-macos=2.8.0` pulls protobuf as a downstream dependency. Version 4.21 of the Python protobuf package had a breaking change that makes it incompatible with tensorflow-2.8.0 (see tensorflow/tensorflow#56077). Later patches to Tensorflow limit the version of protobuf to 3.20. We've got a work item to update the tensorflow we include, but until then this change gives the ceiling on tensorflow's protobuf dependency that its later versions enforce. * chore: update detectron2 example to use v0.6 and reenable nightly test [MLG-301] (#6103) * Run model in EventStorage context * Use new Docker images * Remove pytest.skip from test_detectron2_coco_pytorch_const * Update README.md * Minor code reduction * Dockerfile (listed in .detignore) * Use determinedai repo instead of a personal repo * Makefile for building and publishing the Docker image * docs: Bring content changes from docusaurus-ls beta (#6121) * docs: Bring content changes from docusaurus-ls beta Bring over content changes from the beta including reorganization changes. * additional organizational edits updating index pages, adding a top nav to welcome page * added redirects * revisions based on feedback * rstfmt run * feat: display workspace icon in ProjectCard (#6125) * fix: checkpoint GC should set resource pool (#6136) [DET-9018] * docs: bump rstfmt version (#6138) * chore: add dev cli option to get auth token (#6008) add a `curl` option to help with curling various endpoints * build(deps): bump golang.org/x/net from 0.0.0-20210405180319-a5a99cb37ef4 to 0.7.0 in /proto (#6130) Bumps [golang.org/x/net](https://github.com/golang/net) from 0.0.0-20210405180319-a5a99cb37ef4 to 0.7.0. - [Release notes](https://github.com/golang/net/releases) - [Commits](https://github.com/golang/net/commits/v0.7.0) * perf: Improved performance of historical allocation task endpoint, removed training/validation times (#6135) * fix: FOUNDENG-438 Podman tests from the gate are breaking znodes again (#6146) * chore: add Toggle component to UI Kit [WEB-841] (#6144) * chore: Add tags to UI kit [WEB-816] (#6100) * chore: Move SelectFilter into kit folder and update it [WEB-843] (#6102) * fix: replace `InlineEditor` with UIKit input (#6082) * fix: replace `InlineEditor` * fix: add modal for experiment name * fix: layout of settings page * fix: setting page * fix: minor changes * feat: move experiment `description` and `tags` into edit modal * chore: add `N/A` when description is empty in experiment detail * fix: value bug * fix: revert tag; remove tag from edit modal due to design inconsistancy * chore: add/test pt-only images and bumpenvs (#6097) * add pt images to some unit tests * add pt-images to circleci config * run bumpenvs procedure * fix test function signatures and linting * fix warnings linting * fix docs * expand unit tests coverage * build(deps): bump github.com/prometheus/client_golang from 1.10.0 to 1.11.1 in /master (#6004) Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 1.10.0 to 1.11.1. - [Release notes](https://github.com/prometheus/client_golang/releases) - [Changelog](https://github.com/prometheus/client_golang/blob/main/CHANGELOG.md) - [Commits](prometheus/client_golang@v1.10.0...v1.11.1) * feat: add det.import_from_path (#5737) import_from_path allows users to import arbitrary code from a checkpoint, even if the modules in the checkpoint have the same name as modules they already have imported, but contain different code. This is common when importing, for example, an old model_def.py that has been updated since the original checkpoint was saved. * fix: post rank_id correctly for fluent-less logging (#6151) [DET-8999] * ci: send Slack notification on GKE node pool creation failure (#6152) In order to prevent quota failures from showing up as CI failures, this makes node pool creation failure send a Slack notification and mark the job as successful. I couldn't figure out how to use the Slack orb while distinguishing this particular situation from the general failure case, so I just slapped in a direct request to the already-configured Slack webhook for sending messages to #ci-bots. `circleci-agent step halt` marks the job as successful, which is why we want a notification at all. For some reason, CircleCI fails to provide an equivalent for marking the job as canceled or some other state besides success/failure; we could make a call to the CircleCI API to cancel the current job, but that would rely on having a CircleCI token available, which we're trying to get away from. * chore: drop unused columns from `raw_steps`, `raw_validations`, and `raw_checkpoints`. (#6110) * fix: render spinner while auth check pending (#6098) * chore: update hpc-launching-architecture doc - add default slurm option --no-requeue (#6141) * docs: Content updates (#6154) formatted the setup cluster table to match the approved version in the docusaurus ls beta * feat: display user id in `det user list`. (#6156) * fix: Additional tables get experiment- / workspace-specific storagePath [WEB-962] (#6128) * fix: Additional tables get experiment- and workspace-specific storagePath * useMemo * fix: selection width in `move experiment` modal (#6149) * fix: selection width in `move experiment` modal * fix: add form wrapping * chore: remove change * feat: show trial hyperparameters for custom searchers [MLG-343] (#6162) * feat: show trial hyperparameters for custom searchers [MLG-343] * fix: corrected timestamp handling to do an interval overlap instead of contains (#6164) * chore: add release notes (#6167) * chore: add release notes * format with rstfmt * chore: suppress help output for det dev (#6145) avoid showing the `dev` option in `det -h` output * chore: lock api state for backward compatibility check * chore: bump version: 0.20.0-dev0 -> 0.20.1-dev0 * fix: separate Router and authCheck (#6170) * fix: useMemo does not depend on trial having been loaded (#6173) * chore: pass Labels/project/workspace to TaskSpec (#6172) * refactor: replace user store with observables [WEB-799] (#6140) * fix: hide Foldable menu options when button is visible (#6178) This fixes an issue where, when using a `PageHeaderFoldable` component, options that appear in the header always appear in the overflow menu. * feat: add labels to GCP instances created with det deploy gcp [MLG-170] (#6147) * feat: add labels to GCP instances created with det deploy gcp * Changes to mimic det deploy aws --add-tags * Add labels to other resources as well * revert: reflag new chart experience (#6181) * build: eliminate java dependency for typescript swagger bindings (#6139) * fix: Close expiriment fork/ continue trial modal properly (#6174) * fix: Continue Trial flow does not take the new `max_length` (#6168) * fix: pass workspace ID when creating tensor board from WebUI [WEB-1019] (#6186) * fix: don't print ':' when err msg is empty (#6190) * fix: exp move modal (#6183) * fix: exp move modal * fix: minor fixes * fix: add `archived` param and simplify query (#6175) * fix: add `archived` param and simplify query * chore: indent --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Jerry J. Harrow <84593277+jerryharrow@users.noreply.github.com> Co-authored-by: Nick Doiron <nick.doiron@hpe.com> Co-authored-by: NicholasBlaskey <nick.blaskey@hpe.com> Co-authored-by: Eric <31023784+eecsliu@users.noreply.github.com> Co-authored-by: Keita Nonaka <keita.nonaka@hpe.com> Co-authored-by: Erik Wilson <erik.wilson@hpe.com> Co-authored-by: Hamid Zare <12127420+hamidzr@users.noreply.github.com> Co-authored-by: johnkim-det <97752292+johnkim-det@users.noreply.github.com> Co-authored-by: Ryan <rb@hpe.com> Co-authored-by: Danny Zhu <dzhu@hpe.com> Co-authored-by: CanmingCobble <107056780+CanmingCobble@users.noreply.github.com> Co-authored-by: szewaiyuen6 <sze-wai.yuen@hpe.com> Co-authored-by: Bradley Laney <bradley.laney@hpe.com> Co-authored-by: julian-determined-ai <103522725+julian-determined-ai@users.noreply.github.com> Co-authored-by: Corban Beaird <corban.beaird@hpe.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: liamcli <liam@determined.ai> Co-authored-by: Ashton G <ashton.galloway@hpe.com> Co-authored-by: thiagodallacqua-hpe <104855841+thiagodallacqua-hpe@users.noreply.github.com> Co-authored-by: Max <max.russell@hpe.com> Co-authored-by: Maksim <maksim.kouznetsov@hpe.com> Co-authored-by: Emily <15078396+EmilyBonar@users.noreply.github.com> Co-authored-by: Ilia Glazkov <ilia.glazkov@hpe.com> Co-authored-by: Caleb Hoyoul Kang <caleb.kang@hpe.com> Co-authored-by: Wes Turner <wesley.turner@hpe.com> Co-authored-by: Daniel R. Hunter <103537968+drh-determined-ai@users.noreply.github.com> Co-authored-by: Tara Charter <tara.charter@hpe.com> Co-authored-by: rcorujo <90728398+rcorujo@users.noreply.github.com> Co-authored-by: Guangqing Tang <40620519+gt2345@users.noreply.github.com> Co-authored-by: MikhailKardash <mikhail.kardash@hpe.com> Co-authored-by: Jagadeesh Madagundi <jagadeesh545@gmail.com> Co-authored-by: gt2345 <gt2345@columbia.edu> Co-authored-by: Trent Watson <trent.watson@hpe.com>
- Loading branch information
34 people
committed
Mar 9, 2023
1 parent
9563274
commit 05ee6a2