Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Multi-GPU #2870

Merged
merged 9 commits into from
Aug 13, 2015
Merged

Multi-GPU #2870

merged 9 commits into from
Aug 13, 2015

Commits on Aug 9, 2015

  1. Add BlockingQueue for inter-thread communication

    @cypof @shelhamer
    cypof authored and shelhamer committed Aug 9, 2015
    Configuration menu
    Copy the full SHA
    d94ca3f View commit details
    Browse the repository at this point in the history

  2. Thread-local Caffe

    @cypof @shelhamer
    cypof authored and shelhamer committed Aug 9, 2015
    Configuration menu
    Copy the full SHA
    45d792e View commit details
    Browse the repository at this point in the history

  3. Change the way threads are started and stopped 

    - Interrupt the thread before waiting on join
- Provide a method for looping threads to exit on demand
- CHECK if start and stop succeed instead of returning an error
    @cypof @shelhamer
    cypof authored and shelhamer committed Aug 9, 2015
    Configuration menu
    Copy the full SHA
    73b3d13 View commit details
    Browse the repository at this point in the history

  4. Persistent prefetch thread

    @cypof @shelhamer
    cypof authored and shelhamer committed Aug 9, 2015
    1 Configuration menu
    Copy the full SHA
    ddcdc9d View commit details
    Browse the repository at this point in the history

  5. Add DataReader for parallel training with one DB session 

    - Make sure each solver accesses a different subset of the data
- Sequential reading of DB for performance
- Prefetch a configurable amount of data to host memory
- Distribute data to solvers in round-robin way for determinism
    @cypof @shelhamer
    cypof authored and shelhamer committed Aug 9, 2015
    Configuration menu
    Copy the full SHA
    bcc8f50 View commit details
    Browse the repository at this point in the history

  6. Allocate host memory through cudaMallocHost 

    thanks to discussion by @thatguymike and @flx42
    @cypof @shelhamer
    cypof authored and shelhamer committed Aug 9, 2015
    Configuration menu
    Copy the full SHA
    d2f0457 View commit details
    Browse the repository at this point in the history

  7. Multi-GPU 

    - Parallelize batches among GPUs and tree-reduce the gradients
- The effective batch size scales with the number of devices
- Batch size is multiplied by the number of devices
- Split batches between GPUs, and tree-reduce the gradients
- Detect machine topology (twin-GPU boards, P2P connectivity)
- Track device in syncedmem (thanks @thatguymike)
- Insert a callback in the solver for minimal code change
- Accept list for gpu flag of caffe tool, e.g. '-gpu 0,1' or '-gpu all'.
  Run on default GPU if no ID given.
- Add multi-GPU solver test
- Deterministic architecture for reproducible runs
    @cypof @shelhamer
    cypof authored and shelhamer committed Aug 9, 2015
    Configuration menu
    Copy the full SHA
    e5575cf View commit details
    Browse the repository at this point in the history

  8. Detect topology corner cases and improve broadcast order 

    - Start with distant nodes in broadcast
- Fix outside loop to loop for full tree depth
    @shelhamer
    mhouston authored and shelhamer committed Aug 9, 2015
    Configuration menu
    Copy the full SHA
    335bee7 View commit details
    Browse the repository at this point in the history

  9. [docs] add multi-gpu usage note to interfaces

    @shelhamer
    shelhamer committed Aug 9, 2015
    Configuration menu
    Copy the full SHA
    8771d0f View commit details
    Browse the repository at this point in the history