Skip to content

Commit

Permalink
Use vhosted buckets by default, if available
Browse files Browse the repository at this point in the history
 - We still use the s3.amazonaws.com hostname for the underlying requests,
but for best performance, we want to _connect_ to the vhosted bucket if it's
possible (e.g., valid dns name).
 - We do this by actually looking up DNS when you create a new bucket object
without a conn (and plugging in a modified dialer).

Rationale:
  bucketname.s3.amazonaws.com is geographically cnamed by amazon down through
the appropriate load-balancers and caches to ensure the best performance
of your bulk traffic.

  It also ensures you're pointed to the best /frontend/ to have the fewest
problems with S3's 'eventual consistency' model.

Unanswered Questions:
  - Does Go DNS cache dns results?  If so, does it obey TTL's?

  - What is the meaning of life?
  • Loading branch information
James D. Nurmi committed Jun 22, 2011
1 parent 75b3ecb commit 82e7828
Show file tree
Hide file tree
Showing 2 changed files with 31 additions and 7 deletions.
26 changes: 24 additions & 2 deletions s3/bucket_opers.go
Expand Up @@ -9,6 +9,7 @@ import (
"http"
"io"
"io/ioutil"
"net"
"os"
"path"
"xml"
Expand All @@ -22,16 +23,32 @@ type Bucket struct {
conn *aws.Conn
}


// NewBucket creates a new *Bucket object from an endpoint URL.
//
// URL is the _endpoint_ url (nil will default to https://s3.amazonaws.com/)
// Name should be the bucket name; If you pass this as an empty string, you will regret it.
// conn is OPTIONAL, but allows you to re-use another aws.Conn if you'd like.
//
// If you omit conn, the dialer used will be based off the VHost of your bucket (if possible),
// to ensure best performance (e.g., endpoint associated w/ your bucket, and any
// regional lb's)
func NewBucket(u *http.URL, Name string, conn *aws.Conn)(b *Bucket){
if u == nil {
u = &http.URL{Scheme:"https", Host: USEAST_HOST, Path: "/"}
}
if conn == nil {
conn = aws.NewConn(aws.URLDialer(u, nil))
vname := VhostName(Name, u)
addrs, err := net.LookupHost(vname)
if err == nil && len(addrs) > 0 {
dial_url := &http.URL {
Scheme: u.Scheme,
Host: vname,
Path: u.Path,
}
conn = aws.NewConn(aws.URLDialer(dial_url, nil))
} else {
conn = aws.NewConn(aws.URLDialer(u, nil))
}
}
b = &Bucket {
URL: u,
Expand All @@ -41,6 +58,11 @@ func NewBucket(u *http.URL, Name string, conn *aws.Conn)(b *Bucket){
return
}

// Returns the vhost name of the bucket (bucket.s3.amazonaws.com)
func VhostName(b string, ep *http.URL)(string){
return b + "." + ep.Host
}

func (self *Bucket)key_url(key string)(*http.URL){
return &http.URL {
Scheme: self.URL.Scheme,
Expand Down
12 changes: 7 additions & 5 deletions s3/service_opers.go
Expand Up @@ -28,13 +28,15 @@ func NewService(url *http.URL)(s *Service){
return
}

// Returns a new *Bucket with the same connection and URL
// data as the Service connection. You MUST have already
// created the bucket in order to make use of the Bucket object.
// Returns a new *Bucket with the same URL data as the Service connection.
// You MUST have already created the bucket in order to make use of the
// Bucket object.
//
// See CreateBucket.
// See CreateBucket to create a new bucket.
func (self *Service)Bucket(name string)(*Bucket){
return NewBucket(self.URL, name, self.conn)
// We deliberately do NOT re-use our conn here, in order to take advantage
// of vhosted bucket DNS perks.
return NewBucket(self.URL, name, nil)
}

func s3Path(bucket, key string)(string){
Expand Down

0 comments on commit 82e7828

Please sign in to comment.