Skip to content
master
Switch branches/tags
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

About adaptive-scheduler

Home: http://github.com/basnijholt/adaptive-scheduler

Package license: BSD-3-Clause

Feedstock license: BSD-3-Clause

Summary: An asynchronous scheduler for Adaptive

Development: https://github.com/basnijholt/adaptive-scheduler

Documentation: http://adaptive-scheduler.readthedocs.io

The Adaptive scheduler solves the following problem, you need to run a few 100 learners and can use >1k cores. ipyparallel and dask.distributed provide very powerful engines for interactive sessions. However, when you want to connect to >1k cores it starts to struggle. Besides that, on a shared cluster there is often the problem of starting an interactive session with ample space available. Our approach is to schedule a different job for each adaptive.Learner. The creation and running of these jobs are managed by adaptive-scheduler. This means that your calculation will definitely run, even though the cluster might be fully occupied at the moment. Because of this approach, there is almost no limit to how many cores you want to use. You can either use 10 nodes for 1 job (learner) or 1 core for 1 job (learner) while scheduling hundreds of jobs. Everything is written such that the computation is maximally local. This means that is one of the jobs crashes, there is no problem and it will automatically schedule a new one and continue the calculation where it left off (because of Adaptive's periodic saving functionality). Even if the central "job manager" dies, the jobs will continue to run (although no new jobs will be scheduled.)

Current build status

Azure
VariantStatus
linux_64_python3.10.____cpython variant
linux_64_python3.7.____73_pypy variant
linux_64_python3.7.____cpython variant
linux_64_python3.8.____cpython variant
linux_64_python3.9.____cpython variant
osx_64_python3.10.____cpython variant
osx_64_python3.7.____73_pypy variant
osx_64_python3.7.____cpython variant
osx_64_python3.8.____cpython variant
osx_64_python3.9.____cpython variant

Current release info

Name Downloads Version Platforms
Conda Recipe Conda Downloads Conda Version Conda Platforms

Installing adaptive-scheduler

Installing adaptive-scheduler from the conda-forge channel can be achieved by adding conda-forge to your channels with:

conda config --add channels conda-forge
conda config --set channel_priority strict

Once the conda-forge channel has been enabled, adaptive-scheduler can be installed with:

conda install adaptive-scheduler

It is possible to list all of the versions of adaptive-scheduler available on your platform with:

conda search adaptive-scheduler --channel conda-forge

About conda-forge

Powered by NumFOCUS

conda-forge is a community-led conda channel of installable packages. In order to provide high-quality builds, the process has been automated into the conda-forge GitHub organization. The conda-forge organization contains one repository for each of the installable packages. Such a repository is known as a feedstock.

A feedstock is made up of a conda recipe (the instructions on what and how to build the package) and the necessary configurations for automatic building using freely available continuous integration services. Thanks to the awesome service provided by CircleCI, AppVeyor and TravisCI it is possible to build and upload installable packages to the conda-forge Anaconda-Cloud channel for Linux, Windows and OSX respectively.

To manage the continuous integration and simplify feedstock maintenance conda-smithy has been developed. Using the conda-forge.yml within this repository, it is possible to re-render all of this feedstock's supporting files (e.g. the CI configuration files) with conda smithy rerender.

For more information please check the conda-forge documentation.

Terminology

feedstock - the conda recipe (raw material), supporting scripts and CI configuration.

conda-smithy - the tool which helps orchestrate the feedstock. Its primary use is in the construction of the CI .yml files and simplify the management of many feedstocks.

conda-forge - the place where the feedstock and smithy live and work to produce the finished article (built conda distributions)

Updating adaptive-scheduler-feedstock

If you would like to improve the adaptive-scheduler recipe or build a new package version, please fork this repository and submit a PR. Upon submission, your changes will be run on the appropriate platforms to give the reviewer an opportunity to confirm that the changes result in a successful build. Once merged, the recipe will be re-built and uploaded automatically to the conda-forge channel, whereupon the built conda packages will be available for everybody to install and use from the conda-forge channel. Note that all branches in the conda-forge/adaptive-scheduler-feedstock are immediately built and any created packages are uploaded, so PRs should be based on branches in forks and branches in the main repository should only be used to build distinct package versions.

In order to produce a uniquely identifiable distribution:

  • If the version of a package is not being increased, please add or increase the build/number.
  • If the version of a package is being increased, please remember to return the build/number back to 0.

Feedstock Maintainers

About

A conda-smithy repository for adaptive-scheduler.

Resources

License

Releases

No releases published

Packages

No packages published