Downloading exercises directly into Pharo #32

macta · 2018-07-28T08:09:41Z

Currently Pharo Exercism uses OSProcess to communicate with Exercism. @bencoman experimented with using a direct http connection and using the exercism protocol. This feels like a cleaner solution as shelling out to the os and parsing the osprocess result is more error prone than getting direct http results.

(We currently have a half way house where we can download directly but not yet upload)

macta · 2018-08-06T21:48:37Z

There is an issue in exercism to document this better: exercism/exercism#4087

bencoman · 2018-08-26T00:29:25Z

Here is my experiment... https://github.com/bencoman/pharogui-exercism
but it seems that was based on the V2 client and needs to be updated for V3...
https://github.com/exercism/website-copy/blob/master/pages/cli_v1_to_v2.md

bencoman · 2018-08-27T02:54:44Z

The first job is to see what its service API looks like. Exercism makes this easy with its commandline tool providing a very useful '--verbose' flag to show the operation of CLI commands e.g. exercism help.

Configure

Get your API key YOUR_TOKEN here... https://exercism.io/my/settings
and configure it like...

$ exercism version
exercism version 3.0.8

$ exercism configure --token=YOUR_TOKEN

Downloading exercises with CLI

Exercises are downloaded for a track with the 'download' command, e.g...

$ exercism --verbose download --track=go --exercise=hello-world

the output of which is...

========================= BEGIN DumpRequest =========================
GET /v1/solutions/latest?exercise_id=hello-world&track_id=go HTTP/1.1
Host: api.exercism.io
Authorization: Bearer YOUR_TOKEN
Content-Type: application/json
User-Agent: github.com/exercism/cli v3.0.8 (windows/amd64)


========================= END DumpRequest =========================


========================= BEGIN DumpResponse =========================
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Cache-Control: max-age=0, private, must-revalidate
Connection: keep-alive
Content-Type: application/json; charset=utf-8
Date: Sun, 26 Aug 2018 01:18:58 GMT
Etag: W/"1df219e0ab0e9451c6a7e865beee24d4"
Referrer-Policy: strict-origin-when-cross-origin
Server: nginx/1.10.3 (Ubuntu)
Set-Cookie: site_context=normal; domain=.exercism.io; path=/
Set-Cookie: _exercism_session=Rw9Px8WcBbOHbaSYFGeBGVvZVFHoXrgY2P3fSePB%2BBrhpNE46nzGREyYSY9nbQdq6WdlNdaDWvdlt8W5Qpm45KAc7%2BGMEux1aUTm%2BBNVmb%2B4OcpTzAjPOiTAbd%2BerIgj561MmWhe8uekK1f1v4g5KdT5Ww2Q%2FTZs2nxYfkKLrAq%2BeMPWrgDTMiwoUnWC%2FuwmsXFqpuZqRZnJ%2BZKCKjLxJ4RTRBTw3BONdwT9n91%2B32W%2F8hcBVbvXp%2F0qUYilDFZThlpG--ysKJgbvbbLaJWCUW--3uFmFaSCgL3we4YtaTumkA%3D%3D; domain=.exercism.io; path=/; HttpOnly
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Frame-Options: SAMEORIGIN
X-Permitted-Cross-Domain-Policies: none
X-Request-Id: c0e32dbe-ed4e-427c-aedb-edb73e5cfbcf
X-Runtime: 0.102885
X-Xss-Protection: 1; mode=block


========================= END DumpResponse =========================


========================= BEGIN DumpRequest =========================
GET /v1/solutions/5965748a44ee4ffdb66adce1fe17cd1e/files/README.md HTTP/1.1
Host: api.exercism.io
Authorization: Bearer YOUR_TOKEN
Content-Type: application/json
User-Agent: github.com/exercism/cli v3.0.8 (windows/amd64)


========================= END DumpRequest =========================


========================= BEGIN DumpResponse =========================
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Cache-Control: max-age=0, private, must-revalidate
Connection: keep-alive
Content-Type: text/plain; charset=utf-8
Date: Sun, 26 Aug 2018 01:19:01 GMT
Etag: W/"ae1a1e98b459fe4d3d93c946a66e8191"
Referrer-Policy: strict-origin-when-cross-origin
Server: nginx/1.10.3 (Ubuntu)
Set-Cookie: site_context=normal; domain=.exercism.io; path=/
Set-Cookie: _exercism_session=5PRkw2x%2B3rYtGrE7iePj2kPivNG2Wb8M8ePQzsAAeyadjrFnGs5UCBwu2%2Bbf13R9h2MlOefcOvK2h3BFeIH45qy4AbIcH91Rk5%2B1euzX6Gk9v49j6YAdVpOa68eFNkwE1fd75G0wdpgPZV9Awb9LDFbHy0sNXFxpWAB3lDwvwv9ZiUGW7GDP4hwLyzfhGcvv0fFZIKEqo%2BiSYFkod47QoVQz0KdA5sYkEruXIMsD4OZvpiUaUVv6TB06BVTT%2B3OWTgVOCQ%3D%3D--5t1ThDXrssPwwCv9--Tw2Z%2Ba0M9IwISttJ%2FyZ99g%3D%3D; domain=.exercism.io; path=/; HttpOnly
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Frame-Options: SAMEORIGIN
X-Permitted-Cross-Domain-Policies: none
X-Request-Id: 14f78d42-b2b9-420a-85a2-8ca48176949a
X-Runtime: 0.086310
X-Xss-Protection: 1; mode=block


========================= END DumpResponse =========================


========================= BEGIN DumpRequest =========================
GET /v1/solutions/5965748a44ee4ffdb66adce1fe17cd1e/files/hello_test.go HTTP/1.1
Host: api.exercism.io
Authorization: Bearer YOUR_TOKEN
Content-Type: application/json
User-Agent: github.com/exercism/cli v3.0.8 (windows/amd64)


========================= END DumpRequest =========================


========================= BEGIN DumpResponse =========================
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Cache-Control: max-age=0, private, must-revalidate
Connection: keep-alive
Content-Type: text/plain; charset=utf-8
Date: Sun, 26 Aug 2018 01:19:02 GMT
Etag: W/"81048abeb1cfcaed36184208bbfc335c"
Referrer-Policy: strict-origin-when-cross-origin
Server: nginx/1.10.3 (Ubuntu)
Set-Cookie: site_context=normal; domain=.exercism.io; path=/
Set-Cookie: _exercism_session=3RDgflzyN2XpyIDoOjNeCdo0%2B%2FFP1WUrHmZHy%2BZy%2Bw9OJZ2cT5n%2FDfZNqNeG%2FTAjWYNZlYH7dsXjBwXA3jfb0Sr6povyphO8VIzSHIGqThhv%2Fm5K67N%2B1CDlzb%2BlgB6M%2BUPUsySibMfK8n4dl7ch0SvRIUbfvCrP%2FHMFvn7rUKFlEJNM0FT2zopk3HS9wXnU7xtII1JKwNMCLuLTNlIkwqBPFg67rNMsMj0LlPzpNFtdQ4W%2FM7VhcuTaHQXAN6m8m7IovXQhDy4%3D--4ZYYafLXXLhkByt6--FIkEkJpfKUk4I6rapQeOTA%3D%3D; domain=.exercism.io; path=/; HttpOnly
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Frame-Options: SAMEORIGIN
X-Permitted-Cross-Domain-Policies: none
X-Request-Id: ee75de9d-62d0-4fe1-9df0-6cb2f4b22df6
X-Runtime: 0.075053
X-Xss-Protection: 1; mode=block


========================= END DumpResponse =========================


========================= BEGIN DumpRequest =========================
GET /v1/solutions/5965748a44ee4ffdb66adce1fe17cd1e/files/hello_world.go HTTP/1.1
Host: api.exercism.io
Authorization: Bearer YOUR_TOKEN
Content-Type: application/json
User-Agent: github.com/exercism/cli v3.0.8 (windows/amd64)


========================= END DumpRequest =========================


========================= BEGIN DumpResponse =========================
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Cache-Control: max-age=0, private, must-revalidate
Connection: keep-alive
Content-Type: text/plain; charset=utf-8
Date: Sun, 26 Aug 2018 01:19:02 GMT
Etag: W/"978a6c3cb23cbdd03fece068ffcddbe8"
Referrer-Policy: strict-origin-when-cross-origin
Server: nginx/1.10.3 (Ubuntu)
Set-Cookie: site_context=normal; domain=.exercism.io; path=/
Set-Cookie: _exercism_session=CWjoWFEEYrHIAn3DWVbKhYUMtNRohSgu18%2Fm9umUDT4SfHfV04OThb1uqsNgI4lE2wBeWybp8HXCT19vAhx58YQK6MppJ2YDk83Pf35vTMOBWZgpvpk3qfqR3Meka1AH1L%2BlJQBKILdf3iZ0nbVU%2FLRz1%2F3BY7SqQd9DXGzQteKYeFVMJyzVv5i%2BlWJxKhyIYJzUUovt2CVNtH0sZWocMg6LqV%2B4gU%2BEl6Vy1PLKtU%2FGteowFhQe0IS2%2FY8AlKjpt%2BTYIW6ndlvg--0CcB6sQdIPFEBg%2BI--L6iAKSb1D7tM5sj%2BFJicOQ%3D%3D; domain=.exercism.io; path=/; HttpOnly
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Frame-Options: SAMEORIGIN
X-Permitted-Cross-Domain-Policies: none
X-Request-Id: 9eae9a62-f7ed-4147-9a65-6b42f8b4bbb7
X-Runtime: 0.077996
X-Xss-Protection: 1; mode=block


========================= END DumpResponse =========================

and now there is a new directory "C:\Users...\Exercism\go\hello-world" holding three downloaded files and a fourth file holding solution metadata.

bencoman · 2018-08-27T03:00:24Z

Experimenting in Pharo Playground with established track, e.g. "Go"

Install NeoJSON

Tools > Catalog Browser... NeoJSON > Install stable version.

Then in Playground...

apiKey:=YOUR_TOKEN.
exerciseId:='hello-world'.
trackId:='go'.
client := ZnClient new 
  http;
  host: 'api.exercism.io';
  path: '/v1/solutions/latest';
  headerAt: 'Authorization' put: 'Bearer ' , apiKey;
  queryAt: 'exercise_id' put: exerciseId;
  queryAt: 'track_id' put: trackId.
(response := client get) inspect.

displays a String==>
'{
  "solution": {
    "id": "5965748a44ee4ffdb66adce1fe17cd1e",
    "url": "https://exercism.io/my/solutions/5965748a44ee4ffdb66adce1fe17cd1e",
    "team": null,
    "user": {
      "handle": "bencoman",
      "is_requester": true
    },
    "exercise": {
      "id": "hello-world",
      "instructions_url": "https://exercism.io/my/solutions/5965748a44ee4ffdb66adce1fe17cd1e",
      "auto_approve": true,
      "track": {
        "id": "go",
        "language": "Go"
      }
    },
    "file_download_base_url": "https://api.exercism.io/v1/solutions/5965748a44ee4ffdb66adce1fe17cd1e/files/",
    "files": [
      "README.md",
      "hello_test.go",
      "hello_world.go"
    ],
    "iteration": null
  }
}'

More advanced NeoJSON can parse that directly into instance variables,
but a quick'n'dirty parsing of that JSON into a Dictionary can be done like this...

solution := (NeoJSONReader fromString: response) at: 'solution'.
solution inspect.

Then we can download the files' contents into a collection of Strings...

fileCache := Dictionary new.
(solution at: 'files') do: [ :filename |
	client path: (solution at: 'file_download_base_url') , filename.
	fileCache at: filename put: client get ].
fileCache inspect.

So that demonstrates using NeoJSON to download Exercism track/exercise into Strings we can work with.

bencoman · 2018-08-27T03:26:52Z

Downloading exercises directly into Pharo

I haven't managed to look at the existing tool yet, and this POC is short enough, so I'm just listing the code here in the issue. Here is some discussion about programatically loading Tonel files.
Considering we will have file contents in Strings rather than files, Tim's example is particularly useful.

Using a Command Pattern, i.e. simple classes, one per CLI command. e.g. exercism help download
MVP is a "command-line like experience" doing this in a Playground...

Exercism configureToken: 'YOUR_CLI_TOKEN'. "from https://exercism.io/my/settings".
ExercismDownload exercise: 'hello-world'.

which is faciliated by the following code (presuming NeoJson installed per last post)...

Object subclass: #Exercism
    instanceVariableNames: ''
    classVariableNames: 'ApiPath ApiToken'
    package: 'ExercismDirectExperiment'

Exercism subclass: #ExercismDownload
    instanceVariableNames: ''
    classVariableNames: ''
    package: 'ExercismDirectExperiment'

Exercism >> initialize
    (ApiPath isNil or: ApiToken isNil) 
        ifTrue: [  self error: 'Not configured. Do...   Exercism configureToken: YOUR_TOKEN' ]

Exercism class >> configureToken: your_CLI_token
    "Get your_CLI_token at https://exercism.io/my/settings"
    ApiToken := your_CLI_token.
    ApiPath := '/v1/solutions/latest'.
   
Exercism class >> newClient
    ^ ZnClient new 
          http;
          host: 'api.exercism.io';
          headerAt: 'Authorization' put: 'Bearer ' , ApiToken

ExercismDownload >> track: trackId exercise: exerciseId
    |client response solution fileCache|
    client := self class newClient
        path: ApiPath;
        queryAt: 'exercise_id' put: exerciseId;
        queryAt: 'track_id' put: trackId.
    response := client get.
    solution := (NeoJSONReader fromString: response) at: 'solution'.
    fileCache := Dictionary new.
    (solution at: 'files') do: [ :filename |
        client path: (solution at: 'file_download_base_url') , filename.
        fileCache at: filename put: client get ].
    self loadFiles: fileCache. 

ExercismDownload >> loadFiles: fileCacheMap 
    fileCacheMap keysAndValuesDo: [ :filename :contents |
        (filename endsWith: '.st') ifTrue: [ 
            |parser|
            parser := TonelParser on: contents readStream.
            parser document do: [:item | item load] 
            ]. 
        (filename = 'README.md') ifTrue: [ self notify: contents "quick hack only" ]
    ]

ExercismDownload class >> track: trackId exercise: exerciseId
    ^ self new track: trackId exercise: exerciseId

ExercismDownload class >> exercise: exerciseId
    ^ self track: 'Pharo' exercise: exerciseId

bencoman · 2018-08-27T03:34:40Z

@macta @samWson I'm not in a position to integrate the above with the existing tool, but I saw comments about OSProcess issues on Windows and managed to divert a few hours to updating my previous experiment to the new Excercism-V2 API. Proof of concept is working with hello-world from the Pharo track.
HTH, cheers -ben

bencoman · 2018-08-27T04:00:48Z

@kytrinyx (for interest only), two posts above shows 45 lines for downloading exercises directly into Pharo.

samWson · 2018-08-27T09:46:20Z

@macta A quick update. I hacked together a ZnClient in the playground and got back a good 200 response from Exercism. It wasn't difficult, just some trial and error. And I've been building this on my Windows machine to be sure.

There may be a problem with getting the exercism authentication key from the users user.json file. I kept on getting errors trying to get the file contents. It could be a file permissions issue or I just don't know how to get the contents from a FileReference yet. So I hard coded the authentication key into the request.

The good news is that I don't think this will be difficult or a hack. I'll play around with this some more tomorrow and hopefully have something to show here soon.

bencoman · 2018-08-28T01:13:00Z

@samWson Try this quick test...

ref := 'C:/full/path/to/user.json' asFileReference.
ref exists inspect.
ref inspect.

More FileSystem/FileReference info here.

samWson · 2018-08-28T09:25:07Z

Thanks @bencoman I realized what my problem was. It helps to read the file and not the directory holding the file.

@bencoman I've only just this moment read your code above in depth. Looks like you have our solution all thought out already.

I'm not in a position to integrate the above with the existing tool

If you want I can try to get this integrated into ExercismManager over the next few days.

macta · 2018-08-28T14:13:44Z

Hi guys - I've not had any spare time, but it sounds like you've made some great progress on this - keep going, and I can pr it. If you leave info here - if some time surfaces I can jump back in and help again.

kytrinyx · 2018-08-28T22:20:36Z

This is super exciting!

macta · 2018-08-29T07:03:36Z

@bencoman - nice work, it fitted in pretty seamlessly. @samWson - wasn't sure if you had actually started to try and integrate (if so, interested in comparing your solution to mine - and of course I had the advantage of writing the initial bits so could spot how to do it quickly).

We now need the equivalent foo for submitting files back to Exercism - and then we can get rid of OSProcess (although it would mean we lose the ability to launch the exercism web page from a pharo menu item - but there might be a simpler way to do that, as OSProcess loads quite a lot of cross platform code - although obviously not enough from the problem we hit)

bencoman · 2018-08-29T17:56:32Z

Downloading/submitting exercises directly with the Exercism site doesn't mean OSProcess needs to be abandoned for other functions - particularly cases where you don't need to wait for the external command to complete, like opening a web browser. However here is a possible alternative (though I've not played with it yet) https://github.com/seandenigris/Pharo-Web-Browser http://forum.world.st/ANN-General-WebBrowser-package-for-Pharo4-and-Pharo-5-onwards-td4889670.html

samWson · 2018-09-02T07:06:49Z

@samWson - wasn't sure if you had actually started to try and integrate (if so, interested in comparing your solution to mine - and of course I had the advantage of writing the initial bits so could spot how to do it quickly).

@macta You had your solution finished before I started. It would have taken me much longer to try an integrate it into what we already had anyway. At the moment I'm going to take a look at the problem of submitting exercises. I'll do much the same approach as before: examine the CLI verbose output, and knock together a ZnClent in the playground to do the same thing.

macta · 2018-09-02T14:08:57Z

That would be awesome, if you can work out the foo - we can quickly integrate that bit too. I suspect submission may be a bit more involved, but hopefully not too much.

samWson · 2018-11-17T06:50:09Z

With issue #96 fixed I think this issue can be closed now.

macta mentioned this issue Aug 25, 2018

Test user experience on Windows 10 #31

Closed

samWson added enhancement New feature or request CLI integration Concerns integration of Pharo with the Exercism CLI labels Aug 28, 2018

samWson mentioned this issue Sep 5, 2018

Submitting exercises requires metadata #95

Closed

This was referenced Sep 6, 2018

Submitting solutions direct from Pharo #96

Closed

configure fails for http protocol (i.e. sans https) exercism/cli#732

Closed

Define a public API exercism/exercism#4087

Closed

macta changed the title ~~Improve Exercism<->Pharo communication (use http protocol)~~ Downloading exercises directly into Pharo Sep 6, 2018

samWson closed this as completed Nov 17, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Downloading exercises directly into Pharo #32

Downloading exercises directly into Pharo #32

macta commented Jul 28, 2018 •

edited

Loading

macta commented Aug 6, 2018

bencoman commented Aug 26, 2018 •

edited

Loading

bencoman commented Aug 27, 2018 •

edited

Loading

bencoman commented Aug 27, 2018 •

edited

Loading

bencoman commented Aug 27, 2018 •

edited

Loading

bencoman commented Aug 27, 2018

bencoman commented Aug 27, 2018

samWson commented Aug 27, 2018 •

edited

Loading

bencoman commented Aug 28, 2018

samWson commented Aug 28, 2018 •

edited

Loading

macta commented Aug 28, 2018

kytrinyx commented Aug 28, 2018

macta commented Aug 29, 2018 •

edited

Loading

bencoman commented Aug 29, 2018 via email

samWson commented Sep 2, 2018

macta commented Sep 2, 2018

samWson commented Nov 17, 2018

Downloading exercises directly into Pharo #32

Downloading exercises directly into Pharo #32

Comments

macta commented Jul 28, 2018 • edited Loading

macta commented Aug 6, 2018

bencoman commented Aug 26, 2018 • edited Loading

bencoman commented Aug 27, 2018 • edited Loading

Configure

Downloading exercises with CLI

bencoman commented Aug 27, 2018 • edited Loading

Experimenting in Pharo Playground with established track, e.g. "Go"

Install NeoJSON

Then in Playground...

bencoman commented Aug 27, 2018 • edited Loading

Downloading exercises directly into Pharo

bencoman commented Aug 27, 2018

bencoman commented Aug 27, 2018

samWson commented Aug 27, 2018 • edited Loading

bencoman commented Aug 28, 2018

samWson commented Aug 28, 2018 • edited Loading

macta commented Aug 28, 2018

kytrinyx commented Aug 28, 2018

macta commented Aug 29, 2018 • edited Loading

bencoman commented Aug 29, 2018 via email

samWson commented Sep 2, 2018

macta commented Sep 2, 2018

samWson commented Nov 17, 2018

macta commented Jul 28, 2018 •

edited

Loading

bencoman commented Aug 26, 2018 •

edited

Loading

bencoman commented Aug 27, 2018 •

edited

Loading

bencoman commented Aug 27, 2018 •

edited

Loading

bencoman commented Aug 27, 2018 •

edited

Loading

samWson commented Aug 27, 2018 •

edited

Loading

samWson commented Aug 28, 2018 •

edited

Loading

macta commented Aug 29, 2018 •

edited

Loading