Mass importing URLs

[mutaku]

``>>> for c in b:
... url = "http://192.168.11.4:8001/url/"+urllib.quote(c[0])
... f = urllib.urlopen(url)
... f.read()
...
'

[mutaku]

Not sure why a bunch didn't get short urls but did return 404.
Might switch to 500 and try again and see.

[mutaku]

\nEnvironment:\n\n\nRequest Method: GET\nRequest URL: http://192.168.11.4:8001/url/https://www.amazon.com/gp/dmusic/mp3/player#latestPurchases/\n\nDjango Version: 1.3 SVN-16009\nPython Version: 2.6.5\nInstalled Applications:\n['django.contrib.auth',\n 'django.contrib.contenttypes',\n 'django.contrib.sessions',\n 'django.contrib.sites',\n 'django.contrib.messages',\n 'django.contrib.staticfiles',\n 'django.contrib.admin',\n 'django.contrib.admindocs',\n 'shortener']\nInstalled Middleware:\n('django.middleware.common.CommonMiddleware',\n 'django.contrib.sessions.middleware.SessionMiddleware',\n 'django.middleware.csrf.CsrfViewMiddleware',\n 'django.contrib.auth.middleware.AuthenticationMiddleware',\n 'django.contrib.messages.middleware.MessageMiddleware')\n\n\nTraceback:\nFile "/home/mugen/programming/django/django/core/handlers/base.py" in get_response\n 111. response = callback(request, _callback_args, *_callback_kwargs)\nFile "/home/mugen/programming/Stumpy/shortener/views.py" in submit\n 55. \t\tc.save()\nFile "/home/mugen/programming/Stumpy/shortener/models.py" in save\n 29. \t\t\tself.createShortURL()\nFile "/home/mugen/programming/Stumpy/shortener/models.py" in createShortURL\n 16. \t\tstumps.objects.filter(id=theid).update(shorturl=theshorty)\nFile "/home/mugen/programming/django/django/db/models/query.py" in update\n 465. rows = query.get_compiler(self.db).execute_sql(None)\nFile "/home/mugen/programming/django/django/db/models/sql/compiler.py" in execute_sql\n 869. cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)\nFile "/home/mugen/programming/django/django/db/models/sql/compiler.py" in execute_sql\n 735. cursor.execute(sql, params)\nFile "/home/mugen/programming/django/django/db/backends/util.py" in execute\n 34. return self.cursor.execute(sql, params)\nFile "/home/mugen/programming/django/django/db/backends/mysql/base.py" in execute\n 86. return self.cursor.execute(query, args)\nFile "/usr/lib/pymodules/python2.6/MySQLdb/cursors.py" in execute\n 166. self.errorhandler(self, exc, value)\nFile "/usr/lib/pymodules/python2.6/MySQLdb/connections.py" in defaulterrorhandler\n 35. raise errorclass, errorvalue\n\nException Type: IntegrityError at /url/https://www.amazon.com/gp/dmusic/mp3/player#latestPurchases/\nException Value: (1062, "Duplicate entry '1V' for key 'shorturl'")\n

[mutaku]

I THINK
it's seeing a as the same as A in the short url column (which has a unique tag on it) and isn't letting it in.

[mutaku]

Yes...
for the shorturl column, if it has 'unique=True' it will see 'a' and 'A' as the same thing (which is not for our base62 scheme) and so won't add.
Taking out out 'unique=True' from the shorturl column fixed it.

We still have 'unique=True' on the hashurl column and should be catching any dupes.

[mutaku]

must use urllib.quote_plus(url) to properly feed in urls or a bunch break