feat: Github import #7

MichaelMure · 2018-08-04T14:12:36Z

In a similar fashion as git-bug, Github expose in its graphql API both compiled data of an issue and a timeline of events.

{
  repository(owner: "MichaelMure", name: "git-bug") {
    issues(first: 10) {
      nodes {
        title
        createdAt
        bodyText
        labels(first: 10) {
          nodes {
            name
          }
        }
        timeline(first: 10) {
          nodes {
            __typename
            ... on IssueComment {
              author {
                login
                avatarUrl
              }
              authorAssociation
              bodyText
              createdAt
            }
            ... on LabeledEvent {
              createdAt
              actor {
                login
              }
              label {
                color
                name
              }
            }
            ... on UnlabeledEvent {
              createdAt
              actor {
                login
              }
              label {
                color
                name
              }
            }
          }
        }
      }
    }
  }
}

We can use this timeline to import issue into git-bug. However, there is some mismatch that need to be resolved. Notably, in git-bug, a person is:

type Person struct {
	Name  string
	Email string
}

Where for Github, an actor is:

avatarUrl (URI!)
A URL pointing to the actor's public avatar.

login (String!)
The username of the actor.

resourcePath (URI!)
The HTTP path for this actor.

url (URI!)
The HTTP URL for this actor.

The text was updated successfully, but these errors were encountered:

MichaelMure · 2018-08-12T19:14:02Z

The same timeline is also available through the "normal" API v3: https://developer.github.com/v3/issues/timeline/

That said, 2+ years after its introduction, it's still marked as preview.

The API to get issue timeline events is currently available for developers to preview. During the preview period, the APIs may change without advance notice. Please see the blog post for full details.

geoah · 2018-08-17T14:00:21Z

A bot/plugin would be nice that could sync tickets across git-bug and github.
This would allow devs to always have the issues close to the code and non-devs to be able to create and respond to tickets in github's ui.

aaronsky · 2018-08-18T12:42:14Z

It would be useful to be able to sync tickets from an arbitrary bug tracker by plugging into the backend of git-bug. Something like this would be very useful to me, save for the fact I do not use GitHub.com at work.

MichaelMure · 2018-08-18T18:06:08Z

It would be useful to be able to sync tickets from an arbitrary bug tracker by plugging into the backend of git-bug. Something like this would be very useful to me, save for the fact I do not use GitHub.com at work.

Yes, it should be designed in a generic way so we could support any bugtracker. Github is just the obvious first target.

anarcat · 2018-08-21T15:17:10Z

This would be an absolutely killer feature for me, as it could mean the ability to work on GitHub bugs completely offline, if syncs were two-way and incremental. ;) Obviously, an inflatable raptor would be nice too, but this could be a good first step. Thanks, in any case, for this awesome project, nice to see new blood in that area.

j-f1 · 2018-08-21T16:00:12Z

git-bug could display author names as “[Real Name] (login)” or just the login if the real name is not present or is the same as the login.

anarcat · 2018-08-22T12:29:36Z

yeah, i was also thinking that the Email field in git-bug could be generalized to a URL instead, which would encompass GitHub, but also other platforms...

MichaelMure · 2018-09-24T17:33:32Z

I made some decent progress here:

framework to implement a bridge (importer, exporter or both) + configuration
new commands to configure, list and run an importer/exporter
half of a github importer with currently the process to create and store an API token, and most of the graphql query to fetch the data

MichaelMure · 2018-09-25T11:24:59Z

On a side note, for a proper support for github, #52 needs to be implemented as well ..

MichaelMure · 2018-09-25T16:36:18Z

🎉

MichaelMure · 2018-10-02T11:20:12Z

git-bug core now support comment edition
the importer is now incremental, no need to wipe the bugs anymore, only the new changes will be imported
the importer now import comments edition

So this is .. done ? I would love to have some real world testing, any volunteer ?

MichaelMure · 2018-10-02T11:30:25Z

Some things I purposely leave for the future:

there is room left for optimization. Currently, all the issue data is queried and imported if needed. For this repository, this amount to 49 HTTP requests and 17 seconds. That's decent but too much to use git-bug as a local cache for a github project. I guess there is a clever way to detect early what as changed and prune most of these request.
images/files are not imported, but the core is not fully ready for that yet.

anarcat · 2018-10-02T15:00:34Z

how about gitlab support now? ;)

j-f1 · 2018-10-02T15:14:45Z

You could order the issues by last updated, then pull 10 issues at a time until you get back to the date you last checked @MichaelMure.

MichaelMure · 2018-10-02T15:17:51Z

@anarcat contribution more than welcome ;) I have a limited amount of manpower that will drastically reduce in the following weeks.

If you want to give it a try, implement these interfaces. For reference, the github importer is around 750 lines or code, mainly due to the verbosity of the graphql code.

MichaelMure · 2018-10-02T15:23:52Z

@j-f1 interesting idea. A tricky thing though is the logical clocks. It's better to insert the new operations in the chronological order to have them ordered properly. Admittedly, I'm not doing that very well currently anyway ...

anarcat · 2018-10-02T15:41:59Z

@MichaelMure awesome, glad to see there's an interface and openness to the idea!

anarcat · 2018-10-02T19:36:00Z

So I gave this a shot. A few comments...

I noticed that git-bug already exists as a command on my system. It's part of the git-extras package shipped in Debian, and is a rather silly alias
the setup is a little unintuitive. here's my first try:
```
$ ~/go//bin/git-bug bridge configure 
Building bug cache... Done.
target (github): <HIT ENTER>
invalid target
target (github): ^C
```
Now I may be a little dumb, but I didn't immediately figure out that the word in parenthesis was what I was supposed to write there (and not, as I expected, a default value that would get accepted when I hit enter). When I figured that out, it was more streamlined...
... but then, what do you do with my passphrase?
```
Github project URL: https://github.com/linkchecker/linkchecker/
username: anarcat
password: 
two-factor authentication code
```
it does says above that "git-bug will generate an access token in your Github profile." but it would be better to explicitly state that the password is not written on disk (which, hopefully, it isn't).

then I went into a dance of trying to figure out the right pull incantation. my first attempt failed because of an unrelated error message, which sent me down seeking the magical command:

[1020]anarcat@curie:linkchecker$ ~/go//bin/git-bug bridge pull
import issue: contribution guidelines and FAQ
Error: no diff
Usage:
  git-bug bridge pull [<name>] [flags]

Flags:
  -h, --help   help for pull

[1020]anarcat@curie:linkchecker$ ~/go//bin/git-bug bridge pull default
Error: bad bridge fullname: default
Usage:
  git-bug bridge pull [<name>] [flags]

Flags:
  -h, --help   help for pull

[1021]anarcat@curie:linkchecker1$ ~/go//bin/git-bug bridge pull github
Error: bad bridge fullname: github
Usage:
  git-bug bridge pull [<name>] [flags]

Flags:
  -h, --help   help for pull

[1027]anarcat@curie:linkchecker130$ ~/go//bin/git-bug bridge 
github.default
[1028]anarcat@curie:linkchecker$ ~/go//bin/git-bug bridge pull github.default

Phew! I'm not sure what to do about this. Maybe the problem is the first error shouldn't show the usage: I did call the command correctly and only got confused because I was shown the usage and thought I had to specify (and therefore guess) the right target name.

it does not seem to actually work:
```
[1034]anarcat@curie:linkchecker$ ~/go//bin/git-bug ls
[1035]anarcat@curie:linkchecker$ 
```
Trust me, linkcheck has plenty of issues. :) I wish that was right we had zero issues left, but unfortunately it is more likely not syncing the issues back down...

Thanks for the feature though! Can't wait to try it out. :)

MichaelMure · 2018-10-02T21:47:34Z

Awesome feedback, thanks!

1. I noticed that `git-bug` already exists as a command on my system. It's part of the [git-extras](https://github.com/tj/git-extras) package shipped in Debian, and is a rather [silly alias](https://github.com/tj/git-extras/blob/master/bin/git-bug)

Yeah, that's annoying. I've been pondering on renaming the binary to something like gb and leave the git porcelain world. Really not sure here. I would make the git people happy though.

2. the setup is a little unintuitive. here's my first try:
   ```
   $ ~/go//bin/git-bug bridge configure 
   Building bug cache... Done.
   target (github): <HIT ENTER>
   invalid target
   target (github): ^C
   ```
   Now I may be a little dumb, but I didn't immediately figure out that the word in parenthesis was what I was supposed to write there (and not, as I expected, a default value that would get accepted when I hit enter). When I figured that out, it was more streamlined...

Good point. I thought it would be intuitive once more bridge are added (the prompt would be target (github,gitlab):. But I think I should change the prompt entirely and do something like:

target:
[1]: github
[2]: gitlab

3. ... but then, what do you do with my passphrase?
   ```
   Github project URL: https://github.com/linkchecker/linkchecker/
   username: anarcat
   password: 
   two-factor authentication code
   ```
   it does says above that "git-bug will generate an access token in your Github profile." but it would be better to explicitly state that the password is not written on disk (which, hopefully, it isn't).

It's not stored. Agree on that, more explanation is needed.

4. then I went into a dance of trying to figure out the right `pull` incantation. my first attempt failed because of an unrelated error message, which sent me down seeking the magical command:
   ```
   [1020]anarcat@curie:linkchecker$ ~/go//bin/git-bug bridge pull
   import issue: contribution guidelines and FAQ
   Error: no diff
   Usage:
     git-bug bridge pull [<name>] [flags]
   
   Flags:
     -h, --help   help for pull
   
   [1020]anarcat@curie:linkchecker$ ~/go//bin/git-bug bridge pull default
   Error: bad bridge fullname: default
   Usage:
     git-bug bridge pull [<name>] [flags]
   
   Flags:
     -h, --help   help for pull
   
   [1021]anarcat@curie:linkchecker1$ ~/go//bin/git-bug bridge pull github
   Error: bad bridge fullname: github
   Usage:
     git-bug bridge pull [<name>] [flags]
   
   Flags:
     -h, --help   help for pull
   
   [1027]anarcat@curie:linkchecker130$ ~/go//bin/git-bug bridge 
   github.default
   [1028]anarcat@curie:linkchecker$ ~/go//bin/git-bug bridge pull github.default
   ```
   Phew! I'm not sure what to do about this. Maybe the problem is the first error shouldn't show the usage: I _did_ call the command correctly and only got confused because I was shown the usage and thought I had to specify (and therefore guess) the right target name.

Error: no diff should not happen. The github documentation on this query is full of holes, you must have hit a special case I didn't covered properly.

5. it does not seem to actually work:
   ```
   [1034]anarcat@curie:linkchecker$ ~/go//bin/git-bug ls
   [1035]anarcat@curie:linkchecker$ 
   ```
   Trust me, [linkcheck](https://github.com/linkchecker/linkchecker/) has [plenty of issues](https://github.com/linkchecker/linkchecker/issues). :) I _wish_ that was right we had zero issues left, but unfortunately it is more likely not syncing the issues back down...

I'll give a try with this repo. Hopefully I can hit the same bug.

j-f1 · 2018-10-02T22:27:58Z

For logging in, how about redirecting users to https://github.com/settings/tokens/new?scopes=user,repo&description=git-bug%20token? You can alter the description and scopes as needed, but that would eliminate the need to log in on the terminal. Alternatively, you could create an OAuth app and authenticate with that.

MichaelMure · 2018-10-02T23:26:00Z

For logging in, how about redirecting users to https://github.com/settings/tokens/new?scopes=user,repo&description=git-bug%20token? You can alter the description and scopes as needed, but that would eliminate the need to log in on the terminal.

A question of taste I guess. I feel like it would be a bigger UX hurdle and prone to errors. The way I see it is that git-bug is a free software, meaning anyone can check that the auth informations are handled properly. It's here if anyone want to have a look. Having the token creation handled by git-bug also means that it can be automated if needed.

It could be added as an alternative configure method though.

Alternatively, you could create an OAuth app and authenticate with that.

As far as I know, I can't. A Github Oauth app require an Authorization callback URL. That works for a website, not so much for a terminal app. Too bad, I heard the authorization policy is less crappy ...

j-f1 · 2018-10-02T23:52:51Z

You could start a server on localhost, then pass the port as a parameter to the callback URL, which could be a simple static site that would fetch the token and POST it to the local server, then call window.close().

anarcat · 2018-10-03T00:47:55Z

for the record, i loved that i didn't have to fire up a web browser to login. that part is awesome. just reassure me it's doing the right thing with my password.

as for the second point: just make "enter" choose a sane default value (e.g. "github") and i'll be happy.

and WRT git-rev: wow! didn't know about that newsletter, great! and there are way more git-bug-like packages than i thought!!! thanks :)

waldyrious · 2018-10-03T07:04:56Z

and WRT git-rev: wow! didn't know about that newsletter, great! and there are way more git-bug-like packages than i thought!!! thanks :)

Just because that link is a little buried in @MichaelMure's comment, here's the direct URL for the curious (like me!): https://git.github.io/rev_news/2018/09/19/edition-43/

MichaelMure · 2018-10-03T10:31:56Z

@j-f1 the Authorization callback URL is unique, defined on github in the Oauth app configuration. Also the local server would most likely be unreachable from the internet.

MichaelMure · 2018-10-03T11:32:45Z

@anarcat I tried with your repo, and indeed it fails. I think that what happen is that github introduced the comment history earlier this year but these issues are older. Github simply doesn't have the data anymore.

I'll have to change my code to ignore comment edition in these cases.

j-f1 · 2018-10-03T14:30:39Z

@MichaelMure you can pass a custom redirect URL, as long as it starts with the configured URL.

MichaelMure · 2018-10-03T18:40:25Z

@j-f1 that means that a user would have to entrust their credential to a webservice somewhere, that is, unauditable code. Having that directly in git-bug make more sense imho.

MichaelMure · 2018-10-03T18:58:39Z

By the way, the resulting token is stored in the repo git config. Does someone see a reason why it would be a bad idea ?

MichaelMure · 2018-10-03T19:26:15Z

Alright, I changed the first prompt and added more explanation, it should be less prone to error now.

MichaelMure · 2018-10-05T16:07:23Z

@anarcat I fixed the code for the older issue/comment edition (not pushed yet).

I discovered a new little surprise in your repository: you have an issue (linkchecker/linkchecker#60) opened by a now deleted user. In this case, Github just give a null user.

anarcat · 2018-10-05T16:32:12Z

oh wow, and here i thought that @ghost guy was super productive! ;) TIL!

MichaelMure · 2018-10-06T10:02:34Z

@anarcat It's working for your repo now (i.e. not crashing anymore). Could you check if you see something weird ?

anarcat · 2018-10-06T15:09:44Z

On 2018-10-06 03:02:39, Michael Muré wrote: @anarcat It's working for your repo now (i.e. not crashing anymore). Could you check if you see something weird ?

will do, but it will take a few days.

MichaelMure · 2018-12-01T17:30:19Z

Closing the issue, it's working :-)

MichaelMure added the RFC Request for comments label Aug 6, 2018

MichaelMure added the enhancement New feature or request label Aug 13, 2018

MichaelMure added the Core Concerns the core of git-bug label Aug 23, 2018

anarcat mentioned this issue Oct 2, 2018

feat: Github exporter #56

Closed

MichaelMure closed this as completed Dec 1, 2018

feat: Github import #7

feat: Github import #7

Comments

MichaelMure commented Aug 4, 2018

MichaelMure commented Aug 12, 2018

geoah commented Aug 17, 2018

aaronsky commented Aug 18, 2018

MichaelMure commented Aug 18, 2018

anarcat commented Aug 21, 2018

j-f1 commented Aug 21, 2018

anarcat commented Aug 22, 2018

MichaelMure commented Sep 24, 2018

MichaelMure commented Sep 25, 2018

MichaelMure commented Sep 25, 2018

MichaelMure commented Oct 2, 2018

MichaelMure commented Oct 2, 2018

anarcat commented Oct 2, 2018

j-f1 commented Oct 2, 2018

MichaelMure commented Oct 2, 2018

MichaelMure commented Oct 2, 2018

anarcat commented Oct 2, 2018

anarcat commented Oct 2, 2018

MichaelMure commented Oct 2, 2018

j-f1 commented Oct 2, 2018

MichaelMure commented Oct 2, 2018

j-f1 commented Oct 2, 2018

anarcat commented Oct 3, 2018

waldyrious commented Oct 3, 2018

MichaelMure commented Oct 3, 2018

MichaelMure commented Oct 3, 2018 • edited Loading

j-f1 commented Oct 3, 2018

MichaelMure commented Oct 3, 2018

MichaelMure commented Oct 3, 2018

MichaelMure commented Oct 3, 2018

MichaelMure commented Oct 5, 2018

anarcat commented Oct 5, 2018

MichaelMure commented Oct 6, 2018

anarcat commented Oct 6, 2018 via email

MichaelMure commented Dec 1, 2018

MichaelMure commented Oct 3, 2018 •

edited

Loading