Skip to content

Terry-Mao/bfs

master
Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
doc
 
 
 
 
ops
 
 
 
 
 
 
 
 
 
 
 
 

bfs

bfs 是基于facebook haystack 用golang实现的小文件存储系统。



特性

  • 高吞吐量和低延迟
  • 容错性
  • 高效
  • 维护简单

安装

一、安装hbase、zookeeper

  • 参考hbase官网. 安装、启动请查看这里.
  • 参考zookeeper官网. 安装、启动请查看这里.

二、搭建golang、python环境

三、安装gosnowflake

四、部署

1.下载bfs及依赖包

$ go get -u github.com/Terry-Mao/bfs
$ cd /data/apps/go/src/github.com/Terry-Mao/bfs
$ go get ./...

2.安装directory、store、pitchfork、proxy模块(配置文件请依据实际机器环境配置)

$ cd $GOPATH/src/github.com/Terry-Mao/bfs/directory
$ go install
$ cp directory.toml $GOPATH/bin/directory.toml
$ cd ../store/
$ go install
$ cp store.toml $GOPATH/bin/store.toml
$ cd ../pitchfork/
$ go install
$ cp pitchfork.toml $GOPATH/bin/pitchfork.toml
$ cd ../proxy
$ go install
$ cp proxy.toml $GOPATH/bin/proxy.toml

到此所有的环境都搭建完成!

五、启动

$ cd /$GOPATH/bin
$ nohup $GOPATH/bin/directory -c $GOPATH/bin/directory.toml &
$ nohup $GOPATH/bin/store -c $GOPATH/bin/store.toml &
$ nohup $GOPATH/bin/pitchfork -c $GOPATH/bin/pitchfork.toml &
$ nohup $GOPATH/bin/proxy -c $GOPATH/bin/proxy.toml &
$ cd $GOPATH/github.com/Terry-Mao/bfs/ops
$ nohup python runserver.py &

六、测试

  • bfs初始化,分配存储空间,请查看这里
  • 请求bfs,请查看这里

集群

Aaron Swartz

directory

  • directory主要负责请求的均匀调度和元数据管理,元数据存放在hbase,由gosnowflake产生文件key

store

  • store主要负责文件的物理存储

pitchfork

  • pitchfork负责监控store的服务状态、可用性和磁盘状态

proxy

  • proxy作为bfs存储的代理以及维护bucket相关

ops

  • ops作为bfs的后台管理界面,负责分配存储、扩容、压缩等维护工作

API

api文档

更多

About

distributed file system(small file storage) writen in golang.

Resources

Stars

Watchers

Forks

Packages

No packages published

Languages