Dragonfly is an intelligent P2P based image and file distribution system.
Switch branches/tags
Clone or download
lowzj Merge pull request #267 from allencloud/udpate-header
docs: update copyright header of source file
Latest commit 6c485de Dec 19, 2018
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.circleci test: add integration test framework for Dragonfly Nov 22, 2018
.github refactor: github.com/alibaba-> github.com/dragonflyoss Nov 20, 2018
apis docs: fix bad link caused by the change of d7y.io Dec 19, 2018
build docs: update copyright header of source file Dec 19, 2018
client docs: update copyright header of source file Dec 19, 2018
cmd docs: update copyright header of source file Dec 19, 2018
dfdaemon docs: update copyright header of source file Dec 19, 2018
dfget docs: update copyright header of source file Dec 19, 2018
docs docs: fix bad link caused by the change of d7y.io Dec 19, 2018
hack docs: make swagger.yml to define SuperNode's API Oct 31, 2018
src docs: update copyright header of source file Dec 19, 2018
supernode docs: update copyright header of source file Dec 19, 2018
test test: add integration test framework for Dragonfly Nov 22, 2018
vendor refactor: use cobra to implement the dfget client Dec 18, 2018
version docs: update copyright header of source file Dec 19, 2018
.gitignore feature: implement powerClient and finishTask to implement the basic … Dec 17, 2018
.travis.yml chore: fix travis-ci, only building supernode Aug 2, 2018
CHANGELOG.md docs: update readme to add more info Oct 23, 2018
CODE_OF_CONDUCT.md Create CODE_OF_CONDUCT.md Oct 23, 2018
CONTRIBUTING.md linter: add markdown-link-check for dead link in doc Nov 20, 2018
CONTRIBUTORS docs: auto generate pouch cli docs via code Dec 19, 2018
Dockerfile feature: add Dockerfile of dfclient Nov 16, 2018
FAQ.md docs: fix bad link caused by the change of d7y.io Dec 19, 2018
LICENSE Dragonfly Init Commit Nov 17, 2017
MAINTAINERS.md docs: update company name for zhouhaibing089 Nov 21, 2018
README.md docs: fix bad link caused by the change of d7y.io Dec 19, 2018
ROADMAP.md docs: complete the roadmap.md Oct 11, 2018
adopters.md docs: add ecosystem adopters for Dragonfly Nov 5, 2018
wercker.yml docs: use custom git-pages domain Nov 16, 2018

README.md

Dragonfly

Join the chat at https://gitter.im/alibaba/Dragonfly License FOSSA Status GoDoc Go Report Card Build Status CircleCI codecov

Dragonfly

Contents

Introduction

Dragonfly is an intelligent P2P based image and file distribution system. It aims to resolve issues related to low-efficiency, low-success rate and waste of network bandwidth in file transferring process. Especially in large-scale file distribution scenarios such as application distribution, cache distribution, log distribution, image distribution, etc. In Alibaba, Dragonfly is invoked 2 Billion times and the data distributed is 3.4PB every month. Dragonfly has become one of the most important pieces of infrastructure at Alibaba. The reliability is up to 99.9999% (*1).

While container technologies makes devops life easier most of the time, it sure brings a some challenges: the efficiency of image distribution, especially when you have to replicate image distribution on several hosts. Dragonfly works extremely well with both Docker and PouchContainer for this scenario. It also is compatible with any other container formats.

It delivers up to 57 times the throughput of native docker and saves up to 99.5% the out bandwidth of registry(*2).

Dragonfly makes it simple and cost-effective to set up, operate, and scale any kind of files/images/data distribution.

Features

The project is an open source version of the dragonfly and more internal features will be gradually opened.

  • P2P based file distribution: Using P2P technology for file transmission, which can make full use of the bandwidth resources of each peer to improve download efficiency, saves a lot of cross-IDC bandwidth, especially costly cross-board bandwidth
  • Non-invasive support all kinds of container technologies: Dragonfly can seamlessly support various containers for distributing images.
  • Host level speed limit: Many downloading tools(wget/curl) only have rate limit for the current download task, but dragonfly also provides rate limit for the entire host.
  • Passive CDN: The CDN mechanism can avoid repetitive remote downloads.
  • Strong consistency: Dragonfly can guarantee that all downloaded files must be consistent even if users do not provide any check code(MD5).
  • Disk protection and high efficient IO: Precheck Disk space, delay synchronization, write file-block in the best order, split net-read / disk-write, and so on.
  • High performance: Cluster Manager is completely closed-loop, which means, it does not rely on any DB or distributed cache, processing requests with extremely high performance.
  • Exception auto isolation: Dragonfly will automatically isolate exception nodes(peer or Cluster Manager) to improve download stability.
  • No pressure on file source: Generally, as long as a few Cluster Managers download file from the source.
  • Support standard http header: Support http header, Submit authentication information through http header.
  • Effective concurrency control of Registry Auth: Reduce the pressure of the Registry Auth Service.
  • Simple and easy to use: Very few configurations are needed.

Comparison

Test Environment
Dragonfly server 2 * (24core 64GB 2000Mb/s)
File Source server 2 * (24core 64GB 2000Mb/s)
Client 4core 8GB 200Mb/s
Target file size 200MB
Executed Date 2016-04-20

Performance

For Dragonfly, no matter how many clients issue the file downloading, the average downloading time is always around 12 seconds. And for wget, the downloading time keeps increasing when you have more clients, and as the amount of wget clients reaches 1200, the file source will crash, then it can not serve any client.

Roadmap

For more details about roadmap, please refer to file ROADMAP.md.

Community

You are encouraged to communicate most things via GitHub issues or pull requests.

Other active channels:

Contributing

You are warmly welcomed to hack on Dragonfly. We have prepared a detailed guide CONTRIBUTING.md.

License

Dragonfly is licensed under the Apache License, Version 2.0. See LICENSE for the full license text.