Skip to content

Loading…

Various bugfixes and enhancements #5

Open
wants to merge 24 commits into from

2 participants

@slinkp

No description provided.

slinkp and others added some commits
@slinkp slinkp * Add support for organizations
* Allow 201 responses when creating issues
* Work around github's API throttling (by retrying after 60 seconds on a 403 error)
* Close imported closed tickets
b3fa8c4
jfh Added account option. Fixed encoding. Closing issues supported. Wait …
…between requests to avoid API calls limit. Some bug fixes.
20117d1
@slinkp slinkp Cleanup: whitespace, remove commented function, remove unused imports 56d05ff
@slinkp slinkp Merge branch 'master' of https://github.com/jfhovinne/trac2issues
Adds --account option, fixes encoding to utf8, changes format of
component & milestone labels

Conflicts:
	trac2issues.py
985682d
@slinkp slinkp Improved help. Sleep only as much as needed. Report HTTP errors in ch…
…eckProject(). Better errors if account config not found. Better filtering of null-ish values. Avoid empty comments.
6c72bf4
@slinkp slinkp Fix error when a label contains '/' c0a9a8a
@slinkp slinkp Factor out utf8 encoding of POST data 7f6cd3c
@slinkp slinkp docs update 457a453
Jason Gritman customizing the script for our trac migration 145a5e6
Jason Gritman adapt for github v3 api 3557dee
Jason Gritman support an authors.txt file to map trace to github users 201c474
Jason Gritman load closed milestones 6f49b83
Jason Gritman add start number 16d60fa
@3point2 3point2 adding support for ticket type 5a331f0
@3point2 3point2 fix command line option help texts 1ef87e6
@3point2 3point2 fixing readme to reflect added/fixed options 63f007e
@3point2 3point2 documenting simplejson requirement 2dbc9d7
@slinkp slinkp Merge pull request #1 from techcollective/master
a few small improvements
d019e8c
@slinkp slinkp Merge branch 'master' of https://github.com/jgritman/trac2issues
Conflicts:
	trac2issues.py
cede234
@slinkp slinkp Document that we don't fix ticket markup 2f01321
@slinkp slinkp Many Github API v3 fixes:
* Tokens for auth no longer supported
* Fix various API URLs
* Also, better error info on 4xx errors
* Also, allow specifying the authors.txt file, and document it in --help
8fd3033
@slinkp slinkp * Support dumping issues to disk for use with unofficial bulk import …
…process

* Related refactoring
* Support copying trac resolutions to github labels
* Remove -m option now that Github has milestones
* Better usage message
* Convert 'defect' label to 'bug', for github default label compatibility
* Support users being specified via email address - (only works with bulk
  import, not API)
5427036
@slinkp slinkp Work-in-progress: attachments as gists. Out of time sadly 72c6fbe
@slinkp slinkp Don't swallow errors; remove some cruft during issues API push d33c67c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Aug 25, 2010
  1. @slinkp

    * Add support for organizations

    slinkp committed
    * Allow 201 responses when creating issues
    * Work around github's API throttling (by retrying after 60 seconds on a 403 error)
    * Close imported closed tickets
Commits on Feb 10, 2011
  1. Added account option. Fixed encoding. Closing issues supported. Wait …

    jfh committed
    …between requests to avoid API calls limit. Some bug fixes.
Commits on Mar 10, 2011
  1. @slinkp
  2. @slinkp

    Merge branch 'master' of https://github.com/jfhovinne/trac2issues

    slinkp committed
    Adds --account option, fixes encoding to utf8, changes format of
    component & milestone labels
    
    Conflicts:
    	trac2issues.py
  3. @slinkp

    Improved help. Sleep only as much as needed. Report HTTP errors in ch…

    slinkp committed
    …eckProject(). Better errors if account config not found. Better filtering of null-ish values. Avoid empty comments.
  4. @slinkp
  5. @slinkp
  6. @slinkp

    docs update

    slinkp committed
Commits on Dec 9, 2011
  1. customizing the script for our trac migration

    Jason Gritman committed
Commits on Dec 14, 2011
  1. adapt for github v3 api

    Jason Gritman committed
  2. support an authors.txt file to map trace to github users

    Jason Gritman committed
  3. load closed milestones

    Jason Gritman committed
  4. add start number

    Jason Gritman committed
Commits on Jan 17, 2012
  1. @3point2

    adding support for ticket type

    3point2 committed
  2. @3point2
  3. @3point2
  4. @3point2
  5. @slinkp

    Merge pull request #1 from techcollective/master

    slinkp committed
    a few small improvements
Commits on Aug 6, 2012
  1. @slinkp

    Merge branch 'master' of https://github.com/jgritman/trac2issues

    slinkp committed
    Conflicts:
    	trac2issues.py
  2. @slinkp
  3. @slinkp

    Many Github API v3 fixes:

    slinkp committed
    * Tokens for auth no longer supported
    * Fix various API URLs
    * Also, better error info on 4xx errors
    * Also, allow specifying the authors.txt file, and document it in --help
Commits on Aug 13, 2012
  1. @slinkp

    * Support dumping issues to disk for use with unofficial bulk import …

    slinkp committed
    …process
    
    * Related refactoring
    * Support copying trac resolutions to github labels
    * Remove -m option now that Github has milestones
    * Better usage message
    * Convert 'defect' label to 'bug', for github default label compatibility
    * Support users being specified via email address - (only works with bulk
      import, not API)
Commits on Sep 28, 2012
  1. @slinkp
  2. @slinkp
Showing with 554 additions and 146 deletions.
  1. +61 −6 README.textile
  2. +493 −140 trac2issues.py
View
67 README.textile
@@ -1,24 +1,45 @@
h1. Trac to GitHub Issues conversion script
-Imports are now working, including comments
+Imports are now working, including comments, and various optional
+metadata (as labels).
+
+h2. Requirements
+
+simplejson (can be installed with 'pip install simplejson')
h2. Usage
<pre class="console">$ ./trac2issues.py --help
-Usage: trac2issues.py [options]
+Usage: trac2issues.py [options] import|dump
Options:
-h, --help show this help message and exit
-t TRAC, --trac=TRAC Path to the Trac project to export.
+ -a ACCOUNT, --account=ACCOUNT
+ Name of the GitHub Account to import into. (If neither
+ this nor --account is specified, user from your global
+ git config will be used.)
-p PROJECT, --project=PROJECT
Name of the GitHub Project to import into.
-x, --closed Include closed tickets.
+ -y, --type Create a label for the Trac ticket type.
-c, --component Create a label for the Trac component.
- -m, --milestone Create a label for the Trac milestone.
+ -r, --reporter Create a label for the Trac reporter.
-o, --owner Create a label for the Trac owner.
- -r, --reporter Add a comment naming the reporter.
- -u URL, --url=URL The base URL for the trac install (will also link to
- the old ticket in a comment).
+ -n, --resolution Create a label for the Trac ticket resolution.
+ -u URL, --url=URL Base URL for the Trac install (if specified, will
+ create a link to the old ticket in a comment).
+ -g ORGANIZATION, --org=ORGANIZATION
+ Name of GitHub Organization (supercedes --account)
+ -s START, --start=START
+ The trac ticket to start importing at.
+ --authors=FILE File to load user login names from. Each line is space
+ separated like: trac-login github-login
+
+
+ We no longer have an option to create a label from Trac milestones,
+ because Github issues now supports milestones natively.
+
</pre>
<pre class="console">
@@ -28,5 +49,39 @@ sudo ./trac2issues.py \
-u http://bugs.davglass.com/projects/davglass
</pre>
+h2. Limitations
+
+Does not convert many Trac wiki syntax features to Markdown; so, issues or
+comments that use much Trac syntax will look wrong. One thing we
+do support is the "{{{ ... }}}" literal text delimeters.
+
+It's rather slow: GitHub limits us to 60 API calls per second, and
+each issue might take several calls (one to create the issue, one per
+comment to add).
+
+This varies depending on which options you enable; more = slower.
+
+The Github API does not allow assigning the owner or reporter. This is
+why we have options for attaching those as labels. It's a kludge, yes.
+
+Note that if your github repository already has some issues, you can't
+possibly preserve correct issue numbers. You might maybe be able to
+create a new fork to another account, import your issues to the fork,
+rename the original repository, and transfer ownership of the
+fork. Maybe. Untested.
+
+
+h2. Advice
+
+It is highly recommended to create a scratch github repository to run
+your imports to, and do trial runs until you're happy, before doing it
+to your real project repository. This is because it may take you
+several tries to decide exactly which options you want, and verify
+that it's going to work
+Since there is NO way to delete an issue from github issues, any
+issues you create on your real repository will be there forever.
+This is especially important if you care about preserving issue
+numbers; in that case you should also use the -x option to include
+closed issues.
View
633 trac2issues.py 100755 → 100644
@@ -2,59 +2,131 @@
##Script to convert Trac Tickets to GitHub Issues
+import copy
import re, os, sys, time, math, simplejson
-import string, shutil, urllib2, urllib, pprint, simplejson, datetime
+import string, shutil, urllib2, urllib, pprint, base64, json, getpass
+
from datetime import datetime
from optparse import OptionParser
+from time import sleep
##Setup pp for debugging
pp = pprint.PrettyPrinter(indent=4)
+usage = """Usage: %prog [options] action
+action may be one of 'import', 'dump'
+"""
+parser = OptionParser(usage=usage)
-parser = OptionParser()
parser.add_option('-t', '--trac', dest='trac', help='Path to the Trac project to export.')
+parser.add_option('-a', '--account', dest='account', help='Name of the GitHub Account to import into. (If not specified, user from your global git config will be used.)')
parser.add_option('-p', '--project', dest='project', help='Name of the GitHub Project to import into.')
parser.add_option('-x', '--closed', action="store_true", default=False, dest='closed', help='Include closed tickets.')
+parser.add_option('-y', '--type', action="store_true", default=False, dest='type', help='Create a label for the Trac ticket type.')
parser.add_option('-c', '--component', action="store_true", default=False, dest='component', help='Create a label for the Trac component.')
-parser.add_option('-m', '--milestone', action="store_true", default=False, dest='milestone', help='Create a label for the Trac milestone.')
-parser.add_option('-o', '--owner', action="store_true", default=False, dest='owner', help='Create a label for the Trac owner.')
-parser.add_option('-r', '--reporter', action="store_true", default=False, dest='reporter', help='Add a comment naming the reporter.')
-parser.add_option('-u', '--url', dest='url', help='The base URL for the trac install (will also link to the old ticket in a comment).')
+# parser.add_option('-m', '--milestone', action="store_true", default=False, dest='milestone', help='Create a label for the Trac milestone.')
+parser.add_option('-r', '--reporter', action="store_true", default=False, dest='reporter', help='Create a label for the Trac reporter.')
+parser.add_option('-o', '--owner', action="store_true", default=False, dest='owner', help='Create a label for the Trac owner/assignee.')
+parser.add_option('-n', '--resolution', action="store_true", default=False,
+ help="Create a label for the Trac ticket resolution.")
+parser.add_option('-u', '--url', dest='url', help='Base URL for the Trac install (if specified, will create a link to the old ticket in a comment).')
+parser.add_option('-g', '--org', dest='organization', help='Name of GitHub Organization (supercedes --account)')
+parser.add_option('-s', '--start', dest='start', help='The trac ticket to start importing at.')
+parser.add_option('--authors', dest='authors_file', default='authors.txt',
+ help='File to load user login names from. Each line is space-separated like: trac-login github-login')
+# parser.add_option('--patches-gist', default=False,
+# help='Store attached patches as gists and create a comment linking to the gist.')
(options, args) = parser.parse_args(sys.argv[1:])
-
+# Monkeypatch urllib2 to not treat HTTP 20x as an error.
+# Is there a better way to do this?
+def _non_stupid_http_response(self, request, response):
+ code, msg, hdrs = response.code, response.msg, response.info()
+ if code < 200 or code > 206:
+ response = self.parent.error(
+ 'http', request, response, code, msg, hdrs)
+ return response
+
+urllib2.HTTPErrorProcessor.http_response = _non_stupid_http_response
+urllib2.HTTPErrorProcessor.https_response = _non_stupid_http_response
+
+GITHUB_MAX_PER_MINUTE=60
+_last_ran_at = time.time()
+
+def urlopen(*args, **kw):
+ # As per http://develop.github.com/p/general.html they're limiting
+ # to GITHUB_MAX_PER_MINUTE calls per minute.
+
+ # Normally we wait ~1 second between calls to avoid hitting the
+ # rate limit and having to pause a long time. And to be nice.
+ # (By keeping track of when we actually last ran, we avoid sleeping
+ # longer than needed.)
+ global _last_ran_at
+ when_to_run = _last_ran_at + (60.0 / GITHUB_MAX_PER_MINUTE)
+ sleeptime = max(0, when_to_run - time.time())
+ time.sleep(sleeptime)
+ _last_ran_at = time.time()
+
+ try:
+ return urllib2.urlopen(*args, **kw)
+ except urllib2.HTTPError, e:
+ if e.code == 403:
+ # Maybe we recently ran some other script that hit the rate limit?
+ print bold('Permission denied, waiting a minute and trying again once...')
+ time.sleep(61)
+ return urllib2.urlopen(*args, **kw)
+ else:
+ raise
class ImportTickets:
- def __init__(self, trac=options.trac, project=options.project):
+ def __init__(self, trac=options.trac, account=options.account, project=options.project, authors_file=options.authors_file):
self.env = open_environment(trac)
self.trac = trac
+ self.account = account
self.project = project
self.now = datetime.now(utc)
#Convert the timestamp from a float to an int to drop the .0
self.stamp = int(math.floor(time.time()))
- self.github = 'https://github.com/api/v2/json'
+ self.github = 'https://api.github.com'
try:
self.db = self.env.get_db_cnx()
except TracError, e:
print_error(e.message)
self.includeClosed = options.closed
- self.labelMilestone = options.milestone
+ self.labelType = options.type
+ self.authors_file = authors_file
self.labelComponent = options.component
+ self.labelResolution = options.resolution
self.labelOwner = options.owner
self.labelReporter = options.reporter
+ self.start = options.start
self.useURL = False
-
+ self.organization = options.organization
+ self.reqCount = 0
+ self.milestones = {} # Mapping of title -> id.
+ self.contributors = {}
+ self._milestones_created = set()
if options.url:
- self.useURL = "%s/ticket/" % options.url
+ self.useURL = "%s/ticket/" % (options.url.rstrip('/'))
+
+ self.login = self.password = None
+ self.projectPath = '%s/%s' % (self.organization or self.account or self.login, self.project)
-
+ self._typemap = {
+ 'defect': 'bug',
+ # 'enhancement' is same in trac & github... etc.
+ }
+
+ def importAllToGithub(self):
self.ghAuth()
-
self.checkProject()
+ self.milestones = self.loadMilestones()
+ self.contributors = self.loadContributors()
+ self.labels = self.loadLabels()
if self.useURL:
print bold('Does this look like a valid trac url? [y/N]\n %s1234567' % self.useURL)
@@ -62,41 +134,48 @@ def __init__(self, trac=options.trac, project=options.project):
if go[0:1] != 'y':
print_error('Try Again..')
-
+
##We own this project..
- self._fetchTickets()
+ self.importAllTickets()
+
def checkProject(self):
- url = "%s/repos/show/%s/%s" % (self.github, self.login, self.project)
- data = simplejson.load(urllib.urlopen(url))
+ url = "%s/repos/%s" % (self.github, self.projectPath)
+ try:
+ data = simplejson.load(urlopen(url))
+ except urllib2.HTTPError, e:
+ print_error("Could not connect to project at %s, does it exist? %s" % (url, e))
if 'error' in data:
- print_error("%s/%s: %s" % (self.login, self.project, data['error'][0]['error']))
-
+ print_error("%s: %s" % (self.projectPath, data['error'][0]['error']))
def ghAuth(self):
login = os.popen('git config --global github.user').read().strip()
- token = os.popen('git config --global github.token').read().strip()
if not login:
- print_error('GitHub Login Not Found')
- if not token:
- print_error('GitHub Token Not Found')
+ print_error('GitHub Login Not Found: need github.user in your global config')
self.login = login
- self.token = token
+ print "Gitub password for %s" % login
+ self.password = getpass.getpass()
def _fetchTickets(self):
- cursor = self.db.cursor()
-
+ cursor = self.db.cursor()
+
where = " where (status != 'closed') "
if self.includeClosed:
where = ""
- sql = "select id, summary, description, milestone, component, reporter, owner from ticket %s order by id" % where
+ if self.start:
+ if where:
+ where += " and id >= %s" % self.start
+ else:
+ where = ' where id >= %s' % self.start
+
+ sql = "select id, summary, status, description, milestone, component, reporter, owner, type, resolution from ticket %s order by id" % where
cursor.execute(sql)
# iterate through resultset
tickets = []
- for id, summary, description, milestone, component, reporter, owner in cursor:
+ for id, summary, status, description, milestone, component, reporter, owner, type, resolution in cursor:
if milestone:
milestone = milestone.replace(' ', '_')
if component:
@@ -105,18 +184,25 @@ def _fetchTickets(self):
owner = owner.replace(' ', '_')
if reporter:
reporter = reporter.replace(' ', '_')
-
+ if type:
+ type = type.replace(' ', '_')
+
ticket = {
'id': id,
'summary': summary,
+ 'status': status,
'description': description,
'milestone': milestone,
'component': component,
'reporter': reporter,
'owner': owner,
- 'history': []
+ 'history': [],
+ 'status': status,
+ 'type': type,
+ 'resolution': resolution,
}
- cursor2 = self.db.cursor()
+ # Get all comments.
+ cursor2 = self.db.cursor()
sql = 'select author, time, newvalue from ticket_change where (ticket = %s) and (field = "comment")' % id
cursor2.execute(sql)
for author, time, newvalue in cursor2:
@@ -127,95 +213,384 @@ def _fetchTickets(self):
}
ticket['history'].append(change)
+ # TODO: Create gists for attachments, link to them?
+ # # Get all text attachments.
+ # sql = 'select filename, time, description, author from attachment where (id = %s) and (type = "ticket")' % id
+ # cursor2.execute(sql)
+ # for filename, time, descr, author in cursor2:
+ # unused, ext = os.path.splitext(filename)
+ # if ext.lower() not in ('.txt', '.diff', '.patch', '.py'):
+ # print "Skipping attachment %s of unknown type %s" % (filename, ext)
+ # continue
+ # # There is probably a proper trac API for doing this...
+ # # this will do for now.
+ # attachment_path = os.path.join(self.env.path,
+ # 'attachments', 'ticket',
+ # str(id), filename)
+ # if not os.path.exists(attachment_path):
+ # print "OOps, no such file %s" % attachment_path
+ # continue
+ # # Make a gist.
+ # content = file(attachment_path, 'r').read()
+ # response = self.create_gist(descr, filename, content)
+ # # Create a comment linking to the gist.
+ # gist_url = 'x'
+ # attachment_comment = 'Attachment %s (%s) added by %s' % (filename, gist_url, author)
+ # change = {'author': author,
+ # 'time': time,
+ # 'comment': attachment_comment,
+ # }
+ # Sort comments. Ensure time-based order, for attachments too.
+ ticket['history'].sort(key=lambda item: item['time'])
+
tickets.append(ticket)
- print bold('About to import (%s) tickets from Trac to %s/%s.\n%s? [y/N]' % (len(tickets), self.login, self.project, red('Are you sure you wish to continue')))
- go = sys.stdin.readline().strip().lower()
+ return tickets
- if go[0:1] != 'y':
- print_error('Import Aborted..')
+ def prepareIssue(self, info):
+ """Make a github-compatible dictionary representing the issue.
+ """
+ out = {
+ 'title': info['summary'].encode('utf-8'),
+ 'body': markdown_from_trac(info['description']).encode('utf-8'),
+ 'labels': [],
+ }
+ def info_has_key(key):
+ value = info.get(key)
+ if value is not None and value.strip() not in ('(none)', '', 'Unassigned'):
+ return value
+ return False
+
+ if info_has_key('milestone'):
+ title = info['milestone']
+ if title not in self.milestones:
+ # We assume this number isn't there. TODO: make robust.
+ m_id = len(self.milestones) + 1
+ assert m_id not in self.milestones.values()
+ self.milestones[title] = m_id
+ out['milestone'] = self.milestones[title]
+
+
+ if self.labelType and info_has_key('type'):
+ _type = self._typemap.get(info['type'], info['type'])
+ out['labels'].append(_type)
+
+ if self.labelComponent and info_has_key('component'):
+ out['labels'].append(info['component'])
+
+ if self.labelResolution and info_has_key('status') and info_has_key('resolution'):
+ if info['status'] == 'closed':
+ if info['resolution'] != 'fixed': # too boring to include.
+ out['labels'].append(info['resolution'])
+
+ if info_has_key('owner'):
+ owner = info['owner']
+ if owner in self.contributors:
+ owner = self.contributors[owner]
+ out['assignee'] = owner
+ if self.labelOwner:
+ out['labels'].append('@@%s' % owner)
+
+ if info_has_key('reporter'):
+ # Unfortunately github api v3 still has no way to specify the
+ # creator of an issue. Via the API we can work around with a label...
+ if self.labelReporter:
+ out['labels'].append("@@%s" % info['reporter'])
+ # Otherwise, if using the unofficial bulk dump process, we can set it.
+ out['creator'] = self.parse_user(info['reporter'])
+
+ comments = []
+ for i in info['history']:
+ if i['comment']:
+ body = i['comment'].strip().encode('utf-8', 'replace')
+ # Ignore tracback comments for now.
+ if 'class="tracback"' in body:
+ continue
+ comment = {'body': markdown_from_trac(body)}
+ author = i.get('author', 'anonymous').strip()
+ comment['user'] = self.parse_user(author)
+ comments.append(comment)
- #pp.pprint(tickets)
- for data in tickets:
- self.createIssue(data)
+ if self.useURL:
+ comment = "Ticket imported from Trac:\n %s%s" % (self.useURL, info['id'])
+ comment += "\nReported by: %s" % info['reporter']
+ comments.append({'body': comment})
+
+ out['labels'] = list(set([l.encode('utf-8', 'ignore') for l in out['labels']]))
+ # TODO created/closed/modified timestamps.
+ if info['status'] == 'closed':
+ out['state'] = 'closed'
+
+ return out, comments
-
- def createIssue(self, info):
- print bold('Creating issue.')
- out = {
- 'login': self.login,
- 'token': self.token,
- 'title': info['summary'],
- 'body': info['description']
- }
- data = urllib.urlencode(out)
- url = "%s/issues/open/%s/%s" % (self.github, self.login, self.project)
- req = urllib2.Request(url, data)
- response = urllib2.urlopen(req)
+ def parse_user(self, author):
+ """Returns a dict with user and/or email keys.
+ """
+ if '@' in author:
+ email = author.split('<', 1)[-1].split('>', 1)[0]
+ return {'email': email}
+ else:
+ return {'login': author.encode('utf-8', 'replace')}
+
+ def createIssueViaAPI(self, info):
+ """Add an issue via github API."""
+ print bold('Creating issue from ticket %s' % info['id'])
+ out, comments = self.prepareIssue(info)
+
+ for label in out['labels']:
+ # Labels must exist before being assigned to tickets.
+ self.createLabel(label)
+
+ if out.get('milestone'):
+ # Likewise milestones.
+ self.getOrCreateMilestone(out['milestone'])
+
+ # Remove bulk-import format stuff that the API can't deal with
+ issuedata = copy.deepcopy(out)
+ issuedata.pop('creator', None)
+ issuedata.pop('assignee', None)
+
+ url = "%s/repos/%s/issues" % (self.github, self.projectPath)
+ try:
+ response = self.makeRequest(url, issuedata)
+ except:
+ # todo
+ raise
+
ticket_data = simplejson.load(response)
- if 'number' in ticket_data['issue']:
- num = ticket_data['issue']['number']
+ if 'number' in ticket_data:
+ num = ticket_data['number']
print bold('Issue #%s created.' % num)
else:
print_error('GitHub didn\'t return an issue number :(')
- if self.labelMilestone and 'milestone' in info:
- if info['milestone'] != None:
- self.createLabel(num, "M:%s" % info['milestone'])
+ for comment in comments:
+ self.addComment(num, comment)
- if self.labelComponent and 'component' in info:
- if info['component'] != None:
- self.createLabel(num, "C:%s" % info['component'])
+ if info.get('status') == 'closed':
+ self.closeTicket(num)
- if self.labelOwner and 'owner' in info:
- if info['owner'] != None:
- self.createLabel(num, "@%s" % info['owner'])
+ def createLabel(self, name):
+ """Create a label via the API, if it doesn't already exist."""
+ # Can't add a label to a ticket unless it exists, humph.
+ if name in self.labels:
+ return
+ print bold("\tAdding label %s" % (name,))
+ url = "%s/repos/%s/labels" % (self.github, self.projectPath)
+ out = {'name': name,
+ 'color': "FFFFFF"}
- if self.labelReporter and 'reporter' in info:
- if info['reporter'] != None:
- self.createLabel(num, "@@%s" % info['reporter'])
-
- for i in info['history']:
- if i['author']:
- comment = "Author: %s\n%s" % (i['author'], i['comment'])
+ self.makeRequest(url, out)
+ self.labels.add(name)
+
+ def getOrCreateMilestone(self, name_or_number):
+ # TODO: handle state, description, due date.
+ try:
+ num = int(name_or_number)
+ # Reverse lookup, assuming each value is unique
+ # zip(*foo() makes list of keys and list of vals, in matching order.
+ allnames, allnums = zip(*self.milestones.items())
+ if num in allnums:
+ name = allnames[allnums.index(num)]
else:
- comment = i['comment']
-
- self.addComment(num, comment)
+ name = None
+ except (TypeError, ValueError):
+ name = name_or_number
+ num = self.milestones.get(name)
- if self.useURL:
- comment = "Ticket imported from Trac:\n %s%s" % (self.useURL, info['id'])
- self.addComment(num, comment)
-
+ if num in self._milestones_created:
+ return
- def createLabel(self, num, name):
- print bold("\tAdding label %s to issue # %s" % (name, num))
- url = "%s/issues/label/add/%s/%s/%s/%s" % (self.github, self.login, self.project, name, num)
+ # New milestone, create it.
+ url = "%s/repos/%s/milestones" % (self.github, self.projectPath)
out = {
- 'login': self.login,
- 'token': self.token
+ 'title': name
}
- data = urllib.urlencode(out)
- req = urllib2.Request(url, data)
- response = urllib2.urlopen(req)
- label_data = simplejson.load(response)
-
+ response = self.makeRequest(url, out)
+ milestone_data = simplejson.load(response)
+ num = milestone_data['number']
+ self.milestones[name] = num
+ self._milestones_created.add(num)
+ return num
+
+ def loadMilestones(self):
+ milestones = {}
+ self.loadMilestonesForStatus('open', milestones)
+ self.loadMilestonesForStatus('closed', milestones)
+ for m in milestones.values():
+ self._milestones_created.add(m)
+ return milestones
+
+ def loadMilestonesForStatus(self, param, milestones):
+ url = "%s/repos/%s/milestones?state=%s" % (self.github, self.projectPath, param)
+ response = self.makeRequest(url, None)
+ milestones_data = simplejson.load(response)
+ for milestone_data in milestones_data:
+ print 'Found milestone %s' % milestone_data['title']
+ milestones[milestone_data['title']] = milestone_data['number']
+
+ def loadContributors(self):
+ if (os.path.exists(self.authors_file)):
+ with open(self.authors_file) as fd:
+ collaborators = dict(line.strip().split(None, 1) for line in fd)
+ else:
+ collaborators = {}
+ url = "%s/repos/%s/collaborators" % (self.github, self.projectPath)
+ response = self.makeRequest(url, None)
+ collaborators_data = simplejson.load(response)
+ for collaborator_data in collaborators_data:
+ login = collaborator_data['login']
+ collaborators.setdefault(login, login)
+ return collaborators
+
+ def loadLabels(self):
+ url = '%s/repos/%s/labels' % (self.github, self.projectPath)
+ response = self.makeRequest(url, None)
+ labels = [label['name'] for label in simplejson.load(response)]
+ return set(labels)
+
def addComment(self, num, comment):
+ if not comment:
+ print bold("\tSkipping empty comment on issue # %s" % num)
+ return
print bold("\tAdding comment to issue # %s" % num)
- url = "%s/issues/comment/%s/%s/%s" % (self.github, self.login, self.project, num)
+ url = "%s/repos/%s/issues/%s/comments" % (self.github, self.projectPath, num)
+ response = self.makeRequest(url, comment)
+
+ def closeTicket(self, num):
+ url = "%s/repos/%s/issues/%s" % (self.github, self.projectPath, num)
out = {
- 'login': self.login,
- 'token': self.token,
- 'comment': comment
+ 'state': 'closed'
}
- data = urllib.urlencode(out)
- req = urllib2.Request(url, data)
- response = urllib2.urlopen(req)
-
-
+ response = self.makeRequest(url, out)
+
+ def makeRequest(self, url, out):
+ req = urllib2.Request(url) if out is None else urllib2.Request(url, json.dumps(out))
+
+ base64string = base64.encodestring(
+ '%s:%s' % (self.login, self.password))[:-1]
+ authheader = "Basic %s" % base64string
+ req.add_header("Authorization", authheader)
+ # Setting content type explicitly to avoid known bug where github
+ # occasionally barfs on content that includes a '%'.
+ req.add_header('Content-Type', 'application/json')
+ print url
+ #print json.dumps(out)
+ self.reqCount += 1
+ if (self.reqCount % GITHUB_MAX_PER_MINUTE == 0):
+ self.apiLimitExceeded()
+ print "Request no: %s" % (self.reqCount)
+ try:
+ response = urlopen(req)
+ except urllib2.HTTPError, err:
+ if err.code == 403:
+ self.apiLimitExceeded()
+ response = self.makeRequest(url, out)
+ elif err.code >= 400:
+ sys.stderr.write(red("HTTP error!\n"))
+ sys.stderr.write(err.read() + '\n')
+ raise
+ else:
+ raise
+
+ return response
+
+ def apiLimitExceeded(self):
+ # Don't need this if we always use our own urlopen()
+ pass
+ # self.reqCount = 0
+ # print "Sleeping for 60 seconds"
+ # sleep(60)
+
+
+ def importAllTickets(self):
+ tickets = self._fetchTickets()
+ print bold('About to import (%s) tickets from Trac to %s.\n%s? [y/N]' % (len(tickets), self.projectPath, red('Are you sure you wish to continue')))
+ go = sys.stdin.readline().strip().lower()
+
+ if go[0:1] != 'y':
+ print_error('Import Aborted..')
+ #pp.pprint(tickets)
+ for data in tickets:
+ self.createIssueViaAPI(data)
+
+
+ def dumpAllIssues(self, issuedir):
+ """
+ Useful with the bulk-import-issues beta
+ https://gist.github.com/7f75ced1fa7576412901
+ - NOTE this has been discontinued as of August 2012; there
+ may be a similar feature in future, or not.
+ """
+ tickets = self._fetchTickets()
+ for ticket in tickets:
+ i = ticket['id']
+ ticket, comments = self.prepareIssue(ticket)
+ ticket_filename = os.path.join(issuedir, '%s.json' % i)
+ with file(ticket_filename, 'w') as outfile:
+ json.dump(ticket, outfile, indent=1)
+ comments_filename = os.path.join(outdir, 'issues', '%s.comments.json' % i)
+ with file(comments_filename, 'w') as outfile:
+ json.dump(comments, outfile, indent=1)
+
+ def dumpAllMilestones(self, milestonedir):
+ # TODO: handle state, description, due date.
+ for name, number in self.milestones.items():
+ milestone_filename = os.path.join(milestonedir, '%s.json' % number)
+ with file(milestone_filename, 'w') as outfile:
+ json.dump({'title': name}, outfile, indent=1)
+
+ def dumpAll(self, outdir):
+ issuedir = os.path.join(outdir, 'issues')
+ if not os.path.isdir(issuedir):
+ os.makedirs(issuedir)
+ self.dumpAllIssues(issuedir)
+
+ # Have to do milestones second since milestones are discovered
+ # while iterating over issues.
+ milestonedir = os.path.join(outdir, 'milestones')
+ if not os.path.isdir(milestonedir):
+ os.makedirs(milestonedir)
+ self.dumpAllMilestones(milestonedir)
+
+ def create_gist(self, description, filename, content):
+ """
+ POST a new Gist.
+
+ TODO: OAuth to create them for a user.
+ Currently just anonymous.
+ """
+ gist = {
+ "description": description,
+ "public": True,
+ "files": {
+ filename: {
+ "content": content,
+ }
+ }
+ }
+ url = 'https://api.github.com/gists/'
+ return self.makeRequest(url, gist)
+
+
+def markdown_from_trac(text):
+ # Quick hack to convert some notable trac wiki formatting stuff
+ # to equivalent markdown syntax.
+ text = text.replace('{{{', '```')
+ text = text.replace('}}}', '```')
+ return text
+
+def urlencode_utf8(adict):
+ """Ensure dict's values are all utf-8 before urlencoding it.
+ """
+ data = urllib.urlencode(dict([k, v.encode('utf-8')]
+ for k, v in adict.items()))
+ return data
+
+
##Format bold text
def bold(str):
@@ -231,49 +606,27 @@ def print_error(str):
sys.exit(1)
+
if __name__ == "__main__":
- if len(sys.argv) < 2:
- print "For usage: %s --help" % (sys.argv[0])
- print
+ if not (args and options.trac and options.project):
+ print_error("You need at least an action, and the -t and -p options. For usage: %s --help" % (sys.argv[0]))
+
+ os.environ['PYTHON_EGG_CACHE'] = '/tmp/.egg-cache'
+ os.environ['TRAC_ENV'] = options.trac
+ from trac.core import TracError
+ from trac.env import open_environment
+ from trac.util.datefmt import utc
+
+ importer = ImportTickets()
+ if args[0] == 'import':
+ importer.importAllToGithub()
+ elif args[0] == 'dump':
+ try:
+ outdir = args[1]
+ except IndexError:
+ print_error("Dump action needs an output directory specified")
+ sys.exit(1)
+ importer.dumpAll(outdir)
+ print "Your output is in %s" % outdir
else:
- if not options.trac or not options.project:
- print_error("For usage: %s --help" % (sys.argv[0]))
-
- os.environ['PYTHON_EGG_CACHE'] = '/tmp/.egg-cache'
- os.environ['TRAC_ENV'] = options.trac
- from trac.core import TracError
- from trac.env import open_environment
- from trac.ticket import Ticket
- from trac.ticket.web_ui import TicketModule
- from trac.util.text import to_unicode
- from trac.util.datefmt import utc
- ImportTickets()
-
-
-
-'''
- def _fetchTickets(self):
- changetime = self.stamp - (60 * 60 * 24 * 9)
- cursor = self.db.cursor()
- sql = "select id, summary from ticket where (status = 'infoneeded') and (changetime < %i)" % changetime
- cursor.execute(sql)
- result = cursor.fetchall()
- # iterate through resultset
- for record in result:
- print("Expiring Ticket: #%s :: %s :: %s" % (record[0], record[1], self.project))
- ticket = Ticket(self.env, record[0], self.db)
-
- # determine sequence number...
- cnum = 0
- tm = TicketModule(self.env)
- for change in tm.grouped_changelog_entries(ticket, self.db):
- if change['permanent']:
- cnum += 1
-
- ticket['status'] = 'closed'
- ticket['resolution'] = 'expired'
- ticket.save_changes('trac-bot', 'Ticket automatically closed due to no activity.', self.now, self.db, cnum+1)
- self.db.commit()
- tn = TicketNotifyEmail(self.env)
- tn.notify(ticket, newticket=0, modtime=self.now)
-'''
+ print_error("Need to specify a valid action, either dump or import")
Something went wrong with that request. Please try again.