Skip to content
Dragonfly is an intelligent P2P based image and file distribution system.
Go Shell Other
Branch: master
Clone or download
allencloud Merge pull request #1131 from pouchrobot/auto-doc-2019-12-09
docs: auto generate Dragonfly cli/api/contributors docs via code
Latest commit e77febc Dec 9, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.circleci feature: add yamllint tool to format yaml files Dec 7, 2019
.github delete ertra spaces Sep 3, 2019
apis feature: add yamllint tool to format yaml files Dec 7, 2019
client feature:modify client api Dec 3, 2019
cmd fix: unset the default value of totallimit Dec 4, 2019
dfdaemon feature: make the log related parameter can be configured Nov 15, 2019
dfget fix: unset the default value of totallimit Dec 4, 2019
docs docs: auto generate Dragonfly cli/api docs via code Dec 9, 2019
hack cleanup unused scripts and Makefile Dec 1, 2019
pkg Merge pull request #1099 from widy21/master Dec 1, 2019
supernode refactor: handle errors gracefully Nov 19, 2019
test feature: upgrade golang version to 1.12.10 Oct 16, 2019
version version: add error handling Oct 16, 2019
.gitignore fix(version) : add some punctuation Sep 26, 2019
.golangci.yml feature: add yamllint tool to format yaml files Dec 7, 2019
.goreleaser.yml feature: add yamllint tool to format yaml files Dec 7, 2019
.travis.yml feature: add yamllint tool to format yaml files Dec 7, 2019
CHANGELOG.md docs: update readme to add more info Oct 23, 2018
CODE_OF_CONDUCT.md Create CODE_OF_CONDUCT.md Oct 23, 2018
CONTRIBUTING.md feature: upgrade golang version to 1.12.10 Oct 16, 2019
CONTRIBUTORS docs: auto generate Dragonfly cli/api docs via code Dec 2, 2019
Dockerfile feature: upgrade golang version to 1.12.10 Oct 16, 2019
Dockerfile.supernode feature: upgrade golang version to 1.12.10 Oct 16, 2019
FAQ.md fix bad link Dec 6, 2019
LICENSE Dragonfly Init Commit Nov 17, 2017
MAINTAINERS.md docs: add EddieHuang as a maintainer Aug 12, 2019
Makefile cleanup unused scripts and Makefile Dec 1, 2019
README.md fix some typos in README.md Aug 16, 2019
ROADMAP.md docs: update the roadmap doc Aug 19, 2019
adopters.md Update adopters.md Nov 7, 2019
go.mod feature: add supernode disk gc feature Sep 23, 2019
go.sum feature: add supernode disk gc feature Sep 23, 2019

README.md

Dragonfly

Join the chat at https://gitter.im/alibaba/Dragonfly License FOSSA Status GoDoc CII Best Practices Go Report Card Build Status CircleCI codecov

Dragonfly

Note: The master branch may be in an unstable or even broken state during development. Please use releases instead of the master branch in order to get stable binaries.

Contents

Introduction

Dragonfly is an open source intelligent P2P based image and file distribution system. Its goal is to tackle all distribution problems in cloud native scenarios. Currently Dragonfly focuses on being:

  • Simple: well-defined user-facing API (HTTP), non-invasive to all container engines;
  • Efficient: CDN support, P2P based file distribution to save enterprise bandwidth;
  • Intelligent: host level speed limit, intelligent flow control due to host detection;
  • Secure: block transmission encryption, HTTPS connection support.

Dragonfly is now hosted by the Cloud Native Computing Foundation (CNCF) as a Sandbox Level Project. Originally it was born to solve all kinds of distribution at very large scales, such as application distribution, cache distribution, log distribution, image distribution, and so on.

Dragonfly has finished refactoring in Golang. Now versions > 0.4.0 are totally in Golang, while those < 0.4.0 are in Java. We encourage adopters to try Golang version first, since Java versions will be out of support in the next few releases.

Features

In details, Dragonfly has the following features:

  • P2P based file distribution: Using P2P technology for file transmission, which can make full use of the bandwidth resources of each peer to improve download efficiency, saves a lot of cross-IDC bandwidth, especially costly cross-board bandwidth
  • Non-invasive support for all kinds of container technologies: Dragonfly can seamlessly support various containers for distributing images.
  • Host level speed limit: Many downloading tools(wget/curl) only have rate limit for the current download task, but dragonfly also provides rate limit for the entire host.
  • Passive CDN: The CDN mechanism can avoid repetitive remote downloads.
  • Strong consistency: Dragonfly can guarantee that all downloaded files must be consistent even if users do not provide any check code(MD5).
  • Disk protection and high efficient IO: Precheck Disk space, delay synchronization, write file-block in the best order, split net-read / disk-write, and so on.
  • High performance: Cluster Manager is completely closed-loop, which means, it does not rely on any DB or distributed cache, processing requests with extremely high performance.
  • Exception auto isolation: Dragonfly will automatically isolate exception nodes(peer or Cluster Manager) to improve download stability.
  • No pressure on file source: Generally, as long as a few Cluster Managers download file from the source.
  • Support standard http header: Support http header, Submit authentication information through http header.
  • Effective concurrency control of Registry Auth: Reduce the pressure of the Registry Auth Service.
  • Simple and easy to use: Very few configurations are needed.

Comparison

For Dragonfly, no matter how many clients start the file downloading, the average downloading time is almost stable without increasement (12s in experiment, which means it only takes 12s in total for all client to finish downloading file/image).

And for wget, the downloading time keeps increasing when you have more clients. As the number of wget clients reaches 1200 (in following experiment), the file source will crash, then it can not serve any client.

The following table shows the testing environment and the graph shows the comparison result.

Test Environment Statistics
Dragonfly server 2 * (24core 64GB 2000Mb/s)
File Source server 2 * (24core 64GB 2000Mb/s)
Client 4core 8GB 200Mb/s
Target file size 200MB

Performance

Roadmap

For more details about roadmap, please refer to file ROADMAP.md.

Community

You are encouraged to communicate most things via GitHub issues or pull requests.

Other active channels:

Contributing

You are warmly welcomed to hack on Dragonfly. We have prepared a detailed guide CONTRIBUTING.md.

License

Dragonfly is licensed under the Apache License, Version 2.0. See LICENSE for the full license text.

You can’t perform that action at this time.