Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also compare across forks.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also compare across forks.
base fork: GordonDiggs/cataloguais
base: f38ca3b292
...
head fork: GordonDiggs/cataloguais
compare: 64a574fdc4
  • 4 commits
  • 7 files changed
  • 2 commit comments
  • 1 contributor
View
1  Gemfile
@@ -15,4 +15,5 @@ gem 'googlecharts', :git => 'git://github.com/mattetti/googlecharts.git'
group :test do
gem 'rack-test'
gem 'shoulda-context'
+ gem 'turn'
end
View
4 Gemfile.lock
@@ -8,6 +8,7 @@ GEM
remote: http://rubygems.org/
specs:
addressable (2.2.6)
+ ansi (1.4.2)
bond (0.4.1)
bson_ext (1.3.1)
daemons (1.1.4)
@@ -87,6 +88,8 @@ GEM
eventmachine (>= 0.12.6)
rack (>= 1.0.0)
tilt (1.3.3)
+ turn (0.8.3)
+ ansi
tux (0.3.0)
ripl (>= 0.3.5)
ripl-multi_line (>= 0.2.4)
@@ -110,4 +113,5 @@ DEPENDENCIES
sinatra
sinatra-config-file
thin
+ turn
tux
View
4 README.md
@@ -22,7 +22,7 @@ You should also add an `ENV['ADMIN_PASSWORD']` variable if you want to allow edi
The fields for a row are specified in `settings.yml`. You can specify field names in human terms safely (for instance, a field name of "Album Title" will translate to the attribute name of "album_title". You can also refer to fields by their number (so `item.field0` will be equivalent to `item.title`). If you want to re-order fields in the table, you can just change the order in the settings file.
### Testing
-You can run the tests by running `ruby ./cataloguais_test.rb`. Make sure Postgres is running.
+You can run the tests by running `turn ./test/cataloguais_test.rb`. Make sure Postgres is running. You can also measure the performance of several parts of the app with different volumes of items by running `turn ./test/cataloguais_stress_test.rb`.
### Submitting Patches
If you have a feature, or want to fix a bug, I invite you to do so! Please make sure all code in Pull Requests is documented and tested or it may not be accepted immediately.
@@ -49,7 +49,7 @@ On doit aussi créer un `ENV['ADMIN_PASSWORD']` variable si on veut permettre l'
Les fields d'une rangée sont données en `settings.yml`. On peut donner des noms pour les attributs dans la langue de l'homme (par exemple, un nom de "Titre d'album" va etre un attribut de "titre_d_album". On peut aussi accéder aux attributs par leur nombre (`item.field0` va etre equivalent à `item.titre`). Si on veut réorganiser les attributs, on peut juste la faire dans le fichier des options.
### Testing
-On peut executer les tests avec `ruby ./cataloguais_test.rb`. Postgres doit etre commencé.
+On peut executer les tests avec `turn ./test/cataloguais_test.rb`. Postgres doit etre commencé. On peut aussi mesurer la performance de l'applications avec des nombres d'items differents avec `turn ./test/cataloguais_stress_test.rb`.
### Soumettre du code
Si on a du code pour ce projet, on peut soumettre un Pull Request. SVP, vous assurez que tous le code dans votre Pull Request est doccumenté et évalué avant de le soumettre.
View
21 cataloguais.rb
@@ -32,6 +32,11 @@
DataMapper::Logger.new($stdout, :info)
end
+configure :test do
+ ENV['DATABASE_URL'] = 'postgres://localhost/cataloguais_test'
+ DataMapper::Logger.new($stdout, :error)
+end
+
configure :development do
ENV['ADMIN_PASSWORD'] = 'test'
ENV['DATABASE_URL'] = 'postgres://localhost/cataloguais'
@@ -45,7 +50,7 @@
before do
if ENV['ADMIN_PASSWORD'].nil?
- warn "!!! Admin password not set - editing will be enabled by default"
+ warn "!!! Admin password not set - editing will be enabled by default" unless ENV['RACK_ENV'] == 'test'
session['editing_enabled'] = true
end
end
@@ -171,15 +176,15 @@ def get_occurrences
settings.fields.each do |field|
occurrences[field] = {}
- Item.all.each do |item|
- value = item.send(field.robotize)
- if occurrences[field][value]
- occurrences[field][value] += 1
- else
- occurrences[field][value] = 1
- end
+ # use SQL grouping for fast calculation
+ items = DataMapper.repository.adapter.select("SELECT #{field.robotize} as \"col\", COUNT(*) as \"times\" FROM items GROUP BY #{field.robotize} ORDER BY \"times\" desc")
+
+ # copy the items into the occurrences (since they are an array of structs after selection)
+ items.each do |item|
+ occurrences[field][item[:col]] = item[:times]
end
+ # group all the 1s together
if (others = occurrences[field].select{|k,v| v==1}.size) > 1
occurrences[field].delete_if{|k,v| v==1}
occurrences[field]["Other"] = others
View
55 test/cataloguais_stress_test.rb
@@ -0,0 +1,55 @@
+require_relative './test_helper'
+require 'securerandom'
+
+class CataloguaisTest < Test::Unit::TestCase
+ include Rack::Test::Methods
+
+ def app
+ Sinatra::Application
+ end
+
+ # create a specific number of items (default to 500)
+ def reset_db(num_items = 500)
+ DataMapper.repository.adapter.execute("DELETE FROM items")
+ time = Benchmark.measure do
+ Item.transaction do
+ # create a bunch of items
+ num_items.times do
+ item = Item.new
+ settings.fields.each do |field|
+ item.send("#{field.robotize}=", SecureRandom.base64(42))
+ end
+ item.save
+ end
+ end
+ end
+ puts "Created #{num_items} items in #{time.real} seconds"
+ end
+
+ 10.times do |i|
+ context "with #{1000*(i+1)} items" do
+ setup do
+ @num_items = 1000 * (i+1)
+ reset_db(@num_items) unless @num_items == Item.count
+ end
+
+ should "not take too long to do GET /" do
+ time = Benchmark.measure { get '/' }
+ puts "GET / took #{time.real} seconds with #{@num_items} items"
+ end
+
+ should "not take too long to do GET /export" do
+ time = Benchmark.measure { get '/export' }
+ puts "GET /export took #{time.real} seconds with #{@num_items} items"
+ end
+
+ should "not take too long to do GET /graphs" do
+ settings.graph_urls = {}
+ assert settings.graph_urls.empty?
+ time = Benchmark.measure { get '/graphs' }
+ puts "GET /graphs (with occurrence calculation) took #{time.real} seconds with #{@num_items} items"
+ end
+ end
+ end
+
+end
View
18 cataloguais_test.rb → test/cataloguais_test.rb
@@ -1,9 +1,4 @@
-require './cataloguais'
-require 'shoulda-context'
-require 'test/unit'
-require 'rack/test'
-
-set :environment, :test
+require_relative './test_helper'
class CataloguaisTest < Test::Unit::TestCase
include Rack::Test::Methods
@@ -13,7 +8,6 @@ def app
end
# set up the test db
- MongoMapper.database = "cataloguais_test"
Item.all.each { |item| item.destroy }
context "settings" do
@@ -33,9 +27,9 @@ def app
end
end
- context "POST /new/" do
+ context "POST /new" do
setup do
- post '/new/', {:item => { :title => "My favorite album" }}
+ post '/new', {:item => { :title => "My favorite album" }}
@response = JSON.parse(last_response.body)
end
@@ -90,14 +84,14 @@ def app
end
should "remove item" do
- assert_nil Item.find_by_id(@item_id)
+ assert_nil Item.get(@item_id)
end
end
end
context "Item" do
setup do
- @item = Item.new()
+ @item = Item.new
end
should "have the right number of fields and methods" do
@@ -109,7 +103,7 @@ def app
context ".fields" do
should "return the correct fields" do
- assert_equal settings.fields.collect{ |field| field.robotize }, Item.fields
+ assert_equal [:created_at, :updated_at] | settings.fields.collect{ |field| field.robotize.to_sym }, Item.fields
end
end
end
View
12 test/test_helper.rb
@@ -0,0 +1,12 @@
+ENV['RACK_ENV'] = 'test'
+
+require_relative '../cataloguais'
+require 'shoulda-context'
+require 'test/unit'
+require 'rack/test'
+
+Turn.config do |c|
+ c.format = :dotted
+ c.trace = true
+end
+

Showing you all comments on commits in this comparison.

@GordonDiggs
Owner

Interested in how much faster this is? With the original way and 9000 items, a GET /graphs (including this calculation) request took ~23.4457 seconds. With the new way, the same request took ~0.4663 seconds.

@mrb

Very impressive, @GordonDiggs!

Something went wrong with that request. Please try again.