Server code for RNG testing service

Running at rest.randomsanity.org

DesignNotes.txt

@@ -0,0 +1,32 @@
+An infinite number of heuristic/statistical tests could be added; I
+will be much more likely to consider pull requests that add more if
+you can point to somebody else's code on github that screws up
+pseudorandom number generation in a way that your code catches.
+
+The first rough implementation of the duplication detection used a
+really big bloom filter with a very small false positive rate. That
+was fun to code, but it added a lot of complication and was only three
+or four times smaller on disk than the more straightforward solution
+using AppEngine's key/value store.
+
+The performance bottleneck is datastore reads for the duplication
+detection. Testing a 64-byte byte array is 48 reads (and 2 writes).
+If this service becomes very popular and AppEngine costs become an
+issue that is the first place to optimize.
+
+I've done some preliminary testing and benchmarking of an algorithm
+that uses 6 reads and 3 writes but sacrifices detection if the byte
+arrays overlap in fewer than 32 bytes.
+
+
+Possible future work, if there is demand:
+
+Public keys should be globally unique and look random. A tool that
+finds all the public keys on a system and submits them would be
+useful. It would be even more useful if you could safely run the
+tool twice and not get false-positive reports of non-uniqueness
+(if the query was tagged with the name or IP of the machine the
+server could ignore duplicates with the same tag).
+
+
+

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2017 Gavin Andresen
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -1,2 +1,6 @@
-# rngsanity
-Random Number Generator sanity project
+Server-based, random number generator testing service
+
+AppEngine-based service to sanity test what should be
+cryptographically secure random number bytestreams.
+
+See http://www.randomsanity.org/ for documentation and information.

app.yaml

@@ -0,0 +1,6 @@
+runtime: go
+api_version: go1
+
+handlers:
+- url: /.*
+  script: _go_app

code_of_conduct.md

@@ -0,0 +1,51 @@
+# Contributor Code of Conduct
+
+As contributors and maintainers of this project, and in the interest of
+fostering an open and welcoming community, we pledge to respect all people who
+contribute through reporting issues, posting feature requests, updating
+documentation, submitting pull requests or patches, and other activities.
+
+We are committed to making participation in this project a harassment-free
+experience for everyone, regardless of level of experience, gender, gender
+identity and expression, sexual orientation, disability, personal appearance,
+body size, race, ethnicity, age, religion, or nationality.
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery
+* Personal attacks
+* Trolling or insulting/derogatory comments
+* Public or private harassment
+* Publishing other's private information, such as physical or electronic
+  addresses, without explicit permission
+* Other unethical or unprofessional conduct
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+By adopting this Code of Conduct, project maintainers commit themselves to
+fairly and consistently applying these principles to every aspect of managing
+this project. Project maintainers who do not follow or enforce the Code of
+Conduct may be permanently removed from the project team.
+
+This Code of Conduct applies both within project spaces and in public spaces
+when an individual is representing the project or its community.
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting a project maintainer (see below). All
+complaints will be reviewed and investigated and will result in a response that
+is deemed necessary and appropriate to the circumstances. Maintainers are
+obligated to maintain confidentiality with regard to the reporter of an
+incident.
+
+You may send reports to [our Conduct email](mailto:gavinandresen@gmail.com).
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage],
+version 1.3.0, available at
+[http://contributor-covenant.org/version/1/3/0/][version]
+
+[homepage]: http://contributor-covenant.org
+[version]: http://contributor-covenant.org/version/1/3/0/

entropyheader.go

@@ -0,0 +1,19 @@
+package randomsanity
+
+import (
+	"crypto/rand"
+	"encoding/hex"
+	"net/http"
+)
+
+func addEntropyHeader(w http.ResponseWriter) {
+	// This assumes server has a good crypto/rand
+	// implementation. We could memcache an array
+	// that is initialized to crypto/rand but updated
+	// with every request that comes in with random data.
+	var b [32]byte
+	n, err := rand.Read(b[:])
+	if err == nil && n == len(b) {
+		w.Header().Add("X-Entropy", hex.EncodeToString(b[:]))
+	}
+}

index.yaml

@@ -0,0 +1,6 @@
+indexes:
+
+- kind: NotifyViaEmail
+  properties:
+  - name: UserId
+  - name: Address

notify.go

@@ -0,0 +1,221 @@
+package randomsanity
+
+import (
+	"appengine"
+	"appengine/datastore"
+	"appengine/mail"
+	"crypto/rand"
+	"encoding/hex"
+	"fmt"
+	"log"
+	"net/http"
+	"strings"
+	"time"
+)
+
+import netmail "net/mail"
+
+// Code to notify customer when a rng failure is detected
+
+type NotifyViaEmail struct {
+	UserID  string
+	Address string
+}
+
+// Return userID associated with request (or empty string)
+func userID(ctx appengine.Context, id string) (*datastore.Key, error) {
+	// Only pay attention to ?id=123456 if they've done an authentication loop
+	// and are already in the database
+	if len(id) == 0 {
+		return nil, nil
+	}
+	q := datastore.NewQuery("NotifyViaEmail").Filter("UserID =", id).Limit(1).KeysOnly()
+	keys, err := q.GetAll(ctx, nil)
+	if err != nil || len(keys) == 0 {
+		return nil, err
+	}
+	return keys[0], nil
+}
+
+// Register an email address. To authenticate ownership of the
+// address, the server assigns a random user id and emails it.
+// To mitigate abuse, this method is heavily rate-limited per
+// IP and email address
+func registerEmailHandler(w http.ResponseWriter, r *http.Request) {
+	// Requests generated by web browsers are not allowed:
+	if r.Header.Get("Origin") != "" {
+		http.Error(w, "CORS requests are not allowed", http.StatusForbidden)
+		return
+	}
+	ua := r.Header.Get("User-Agent")
+	if len(ua) < 4 || (!strings.EqualFold(ua[0:4], "curl") && !strings.EqualFold(ua[0:4], "wget")) {
+		http.Error(w, "Email registration must be done via curl or wget", http.StatusForbidden)
+		return
+	}
+
+	w.Header().Add("Content-Type", "text/plain")
+	parts := strings.Split(r.URL.Path, "/")
+	if len(parts) < 4 {
+		http.Error(w, "Missing email", http.StatusBadRequest)
+		return
+	}
+	if len(parts) > 4 {
+		http.Error(w, "URL path too long", http.StatusBadRequest)
+		return
+	}
+
+	addresses, err := netmail.ParseAddressList(parts[len(parts)-1])
+	if err != nil || len(addresses) != 1 {
+		http.Error(w, "Invalid email address", http.StatusBadRequest)
+		return
+	}
+	address := addresses[0]
+
+	ctx := appengine.NewContext(r)
+
+	// 2 registrations per IP per day
+	limited, err := RateLimitResponse(ctx, w, IPKey("emailreg", r.RemoteAddr), 2, time.Hour*24)
+	if err != nil || limited {
+		return
+	}
+	// ... and 1 per email per week
+	limited, err = RateLimitResponse(ctx, w, "emailreg"+address.Address, 1, time.Hour*24*7)
+	if err != nil || limited {
+		return
+	}
+	// ... and global 10 signups per hour (so a botnet with lots of IPs cannot
+	// generate a huge surge of bogus registrations)
+	limited, err = RateLimitResponse(ctx, w, "emailreg", 10, time.Hour)
+	if err != nil || limited {
+		return
+	}
+	// Note: the AppEngine dashboard can also be used to set quotas.
+	// If somebody with a bunch of IP addresses is persistently annoying,
+	// we'll switch to a web page with a CAPTCHA or require sign-in with
+	// a Google account to register or require payment to register.
+
+	var notify []NotifyViaEmail
+	q := datastore.NewQuery("NotifyViaEmail").Filter("Address =", address.Address)
+	if _, err := q.GetAll(ctx, &notify); err != nil {
+		http.Error(w, "Datastore error", http.StatusInternalServerError)
+		return
+	}
+	if len(notify) > 0 {
+		sendNewID(ctx, address.Address, notify[0].UserID)
+		fmt.Fprintf(w, "Check your email, ID sent to %s\n", address.Address)
+		return
+	}
+	bytes := make([]byte, 8)
+	if _, err := rand.Read(bytes); err != nil {
+		http.Error(w, "rand.Read error", http.StatusInternalServerError)
+		return
+	}
+	id := hex.EncodeToString(bytes)
+	n := NotifyViaEmail{id, address.Address}
+	k := datastore.NewIncompleteKey(ctx, "NotifyViaEmail", nil)
+	if _, err := datastore.Put(ctx, k, &n); err != nil {
+		http.Error(w, "Datastore error", http.StatusInternalServerError)
+		return
+	}
+	sendNewID(ctx, address.Address, id)
+	// HTTP response MUST NOT contain the id
+	fmt.Fprintf(w, "Check your email, ID sent to %s", address.Address)
+}
+
+// Unregister, given userID
+func unRegisterIDHandler(w http.ResponseWriter, r *http.Request) {
+	if r.Method != "DELETE" {
+		http.Error(w, "unregister method must be DELETE", http.StatusBadRequest)
+		return
+	}
+	parts := strings.Split(r.URL.Path, "/")
+	if len(parts) < 4 {
+		http.Error(w, "Missing userID", http.StatusBadRequest)
+		return
+	}
+	if len(parts) > 4 {
+		http.Error(w, "URL path too long", http.StatusBadRequest)
+		return
+	}
+	ctx := appengine.NewContext(r)
+
+	uID := parts[len(parts)-1]
+	dbKey, err := userID(ctx, uID)
+	if err != nil {
+		http.Error(w, "datastore error", http.StatusInternalServerError)
+		return
+	}
+	if dbKey == nil {
+		http.Error(w, "User ID not found", http.StatusNotFound)
+		return
+	}
+	err = datastore.Delete(ctx, dbKey)
+	if err != nil {
+		http.Error(w, "Error deleting key", http.StatusInternalServerError)
+		return
+	}
+	fmt.Fprintf(w, "id %s unregistered\n", uID)
+}
+
+func sendNewID(ctx appengine.Context, address string, id string) {
+	msg := &mail.Message{
+		Sender:  "randomsanityalerts@gmail.com",
+		To:      []string{address},
+		Subject: "Random Sanity id request",
+	}
+	msg.Body = fmt.Sprintf("Somebody requested an id for this email address (%s)\n"+
+		"for the randomsanity.org service.\n"+
+		"\n"+
+		"id: %s\n"+
+		"\n"+
+		"Append ?id=%s to API calls to be notified of failures via email.\n"+
+		"\n"+
+		"If somebody is pretending to be you and you don't use the randomsanity.org\n"+
+		"service, please ignore this message.\n",
+		address, id, id)
+	if err := mail.Send(ctx, msg); err != nil {
+		log.Printf("mail.Send failed: %s", err)
+	}
+}
+
+func sendEmail(ctx appengine.Context, address string, tag string, b []byte, reason string) {
+	// Don't spam if there are hundreds of failures, limit to
+	// a handful per day:
+	limit, err := RateLimit(ctx, address, 5, time.Hour*24)
+	if err != nil || limit {
+		return
+	}
+
+	msg := &mail.Message{
+		Sender:  "randomsanityalerts@gmail.com",
+		To:      []string{address},
+		Subject: "Random Number Generator Failure Detected",
+	}
+	msg.Body = fmt.Sprintf("The randomsanity.org service has detected a failure.\n"+
+		"\n"+
+		"Failure reason: %s\n"+
+		"Data: 0x%s\n"+
+		"Tag: %s\n", reason, hex.EncodeToString(b), tag)
+	if err := mail.Send(ctx, msg); err != nil {
+		log.Printf("mail.Send failed: %s", err)
+	}
+}
+
+func notify(ctx appengine.Context, uid string, tag string, b []byte, reason string) {
+	if len(uid) == 0 {
+		return
+	}
+	q := datastore.NewQuery("NotifyViaEmail").Filter("UserID =", uid)
+	for t := q.Run(ctx); ; {
+		var d NotifyViaEmail
+		_, err := t.Next(&d)
+		if err == datastore.Done {
+			break
+		}
+		if err != nil {
+			log.Printf("Datastore error: %s", err.Error())
+			return
+		}
+		sendEmail(ctx, d.Address, tag, b, reason)
+	}
+}