Add new store endpoints & snap find support for sections handling #2288

AlexandreAbreu · 2016-11-17T04:43:22Z

Add new store endpoints & snap find support for sections handling.

The sections endpoint is currently in the staging store.

mvo5 · 2016-11-17T16:17:35Z

store/store.go

@@ -885,6 +903,8 @@ func (s *Store) Find(search *Search, user *auth.UserState) ([]*snap.Info, error)
 	} else {
 		q.Set("q", searchTerm)
 	}
+	fmt.Println(search.Section)


Probably something we want to remove before landing.

chipaca

This looks very good! I especially liked that you've implemented a Completer for it.

I'm marking it as "request changes" because of the retry dance; all the rest are trivials.

Good job!

chipaca · 2016-11-21T14:31:56Z

cmd/snap/cmd_find.go

@@ -84,8 +84,22 @@ func getPriceString(prices map[string]float64, suggestedCurrency, status string)
 	return formatPrice(price, currency)
 }

+type SectionName string


chipaca · 2016-11-21T14:32:31Z

cmd/snap/cmd_find.go

@@ -96,20 +110,17 @@ func init() {
 		return &cmdFind{}
 	}, map[string]string{
 		"private": i18n.G("Search private snaps"),
+		"section": i18n.G("Restrict the search to a given section name"),


I'd suggest dropping the name from the string

chipaca · 2016-11-21T14:35:14Z

daemon/api.go

+	case store.ErrEmptyQuery, store.ErrBadQuery:
+		return BadRequest("%v", err)
+	case store.ErrUnauthenticated:
+		return Unauthorized(err.Error())


why ("%v", err) in the others and err.Error() here? AFAIK they're both the same unless err==nil, which you already split out explicitly

chipaca · 2016-11-21T14:37:48Z

daemon/api.go

+		return InternalError("%v", err)
+	}
+
+	results := make([]*json.RawMessage, 0, len(sections))


I'm missing something here. sections is a []string, so why not just return it? As in return SyncResponse(sections, meta)?

chipaca · 2016-11-21T14:42:39Z

store/store.go

+		URL:    &u,
+		Accept: halJsonContentType,
+	}
+	resp, err := s.doRequest(s.client, reqOptions, user)


as this is new code, could you write it already doing the retry dance? Search for retry.Start in this same file for examples. We're slowly rewriting all things that call doRequest to work this way, but we shouldn't add to the backlog.

chipaca · 2016-11-21T14:43:17Z

store/store.go

+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != 200 {
+		return nil, respToError(resp, "search")


it's not really "search", is it?

chipaca · 2016-11-21T14:43:46Z

store/store.go

+	}
+
+	if ct := resp.Header.Get("Content-Type"); ct != halJsonContentType {
+		return nil, fmt.Errorf("received an unexpected content type (%q) when trying to search via %q", ct, resp.Request.URL)


or is it search? I think it isn't search.

chipaca · 2016-11-21T14:44:16Z

store/store.go

+
+	dec := json.NewDecoder(resp.Body)
+	if err := dec.Decode(&sectionData); err != nil {
+		return nil, fmt.Errorf("cannot decode reply (got %v) when trying to get sections via %q", err, resp.Request.URL)


OTOH this one seems right; make the other two like this one :-)

mvo5

Looks good, thank you. One question - it looks like now we do allow a snap find without further arguments again. AIUI this will mean that we get the first 100 snaps again as "featured". But right now AFAICT the store does not support this so we get a random list of 100 snaps (which may include useless ones like test-snapd-python-webserver). If that is indeed the case, should we keep the restriction for snap find for now (until the store has support for featured)?

mvo5 · 2016-11-21T15:02:46Z

client/packages.go

+// GetSections returns the list of existing snap sections in the store
+func (client *Client) GetSections() ([]string, error) {
+	var sections []string
+	_, err := client.doSync("GET", "/v2/sections", nil, nil, nil, &sections)


(nitpick) This and the following line could be combined into a single: if _err := client.doSync(..); err != nil { line. But its not really important (just an idiom we use a lot in the code).

Well maybe but I did it this way to stay consistent with the rest of the code that does not seem to merge assignment/tests,

mvo5 · 2016-11-21T15:04:02Z

cmd/snap/cmd_find.go

@@ -84,8 +84,22 @@ func getPriceString(prices map[string]float64, suggestedCurrency, status string)
 	return formatPrice(price, currency)
 }

+type SectionName string
+
+func (s *SectionName) Complete(match string) []flags.Completion {


mvo5 · 2016-11-21T15:05:51Z

cmd/snap/main.go

@@ -153,7 +153,6 @@ The snap tool interacts with the snapd daemon to control the snappy software pla

 		cmd, err := parser.AddCommand(c.name, c.shortHelp, strings.TrimSpace(c.longHelp), obj)
 		if err != nil {
-


Thanks for catching that.

niemeyer

Looks pretty good, thanks for contributing it.

A few comments:

niemeyer · 2016-11-21T19:54:09Z

client/packages.go

@@ -129,6 +130,16 @@ func (client *Client) List(names []string) ([]*Snap, error) {
 	return result, nil
 }

+// GetSections returns the list of existing snap sections in the store
+func (client *Client) GetSections() ([]string, error) {


s/Get//, please. (client.Snap, client.Changes, etc).

(also maybe interessting in this context in "effective go" which is a interesting read: https://golang.org/doc/effective_go.html#Getters)

niemeyer · 2016-11-21T19:58:48Z

client/packages.go

@@ -150,6 +161,7 @@ func (client *Client) Find(opts *FindOptions) ([]*Snap, *ResultInfo, error) {
 	case opts.Private:
 		q.Set("select", "private")
 	}
+	q.Set("section", opts.Section)


if opts.Section != "" { ... }, so we continue to not send it unless requested? Feels slightly cleaner than sending an empty string which is ignored.

niemeyer · 2016-11-21T20:00:02Z

client/packages_test.go

@@ -43,7 +43,7 @@ func (cs *clientSuite) TestClientFindRefreshSetsQuery(c *check.C) {
 	c.Check(cs.req.Method, check.Equals, "GET")
 	c.Check(cs.req.URL.Path, check.Equals, "/v2/find")
 	c.Check(cs.req.URL.Query(), check.DeepEquals, url.Values{
-		"q": []string{""}, "select": []string{"refresh"},
+		"q": []string{""}, "section": []string{""}, "select": []string{"refresh"},


We need a test where the string is not empty.

niemeyer · 2016-11-21T20:02:14Z

cmd/snap/cmd_find.go

+func (s *SectionName) Complete(match string) []flags.Completion {
+	// TODO cache the result
+	cli := Client()
+	sections, _ := cli.GetSections()


if err != nil { return nil }, more explicit and clear than just ignoring the error silently, which tends to raise eyebrows. Also saves the allocation bellow, but that's an irrelevant saving in this context.

niemeyer · 2016-11-21T20:03:48Z

cmd/snap/cmd_find.go

+type SectionName string
+
+func (s *SectionName) Complete(match string) []flags.Completion {
+	// TODO cache the result


Should be cached on the snapd daemon end rather than here, which means we can cache in memory instead of on disk. So it's okay to drop the TODO here.

niemeyer · 2016-11-21T20:20:59Z

store/store.go

+
+	q := u.Query()
+
+	q.Set("confinement", "strict")


Why is this necessary? We are asking for store sections.. sections don't have confinement settings? Doesn't feel right.

niemeyer · 2016-11-21T20:22:06Z

store/store.go

+
+		return sectionNames, nil
+	}
+	panic("unreachable")


This is bogus. Please catch up with @stolowski.

niemeyer · 2016-11-21T20:28:29Z

store/store_test.go

+*/
+const MockSectionsJSON = `{
+  "_embedded": {
+    "clickindex:sections": [


There's probably not much gain in having 20 lines here instead of:

"clickindex:sections": [{"name": "featured"}, {"name": "database"}],

The line above is still better than the 8 lines below.

niemeyer · 2016-11-21T20:28:43Z

store/store_test.go

+	repo := New(&cfg, nil)
+	c.Assert(repo, NotNil)
+
+	_, err := repo.Sections(t.user)


Shouldn't this be checking the result?

niemeyer · 2016-11-21T20:29:57Z

tests/main/searching/task.yaml

+    expected="(?s)Name +Version +Developer +Notes +Summary *\n\
+    (.*?\n)?\
+    .*"
+    snap find | grep -Pzq "$expected"


Can we test something we know will tend to remain featured here, so we have a hint that the feature in fact works?

mvo5 · 2016-11-23T08:21:51Z

@niemeyer @AlexandreAbreu I would love to get this into 2.18, this is why I picked it up in #2333 and did some minor tweaks (most importantly to use the new store.retryRequest() helper and minor tweaks to respond to Gustavos feedback). Feel free to cherry-pick in here if its not reviewed/merged. If it misses 2.18 thats fine, I think we just continue in here in this case.

One thing (maybe not for this branch) I noticed is that $ snap find --section=invalid-or-misspelled gives a rather ugly http 400 error message right now. It looks like we do not have much logic to decode store errors properly yet (store.go:respToError()). The store sends a 400 error with the body:

{
  "error_list": [
    {
      "code": "invalid-field", 
      "message": "Invalid section given: not-exists"
    }
  ]
}

If we translated that to a go error we could convert it to an error-kind in our REST api and the client could show a nicer error message. But probably a branch on its own :)

small tweaks for the store branch

mvo5 · 2016-11-24T09:09:29Z

@AlexandreAbreu Yeah, the hardcoding of that string is not ideal, I think we want to tweak that in the future.

niemeyer

Looks good, but there's a test missing.

niemeyer · 2016-11-24T14:09:41Z

cmd/snap/cmd_find.go

-		return errors.New(i18n.G("you need to specify a query. Try \"snap find hello-world\"."))
+	// magic! `snap find` returns the featured snaps
+	if x.Positional.Query == "" && x.Section == "" {
+		x.Section = "featured"


This looks untested.

niemeyer · 2016-11-24T14:10:14Z

store/store_test.go

+*/
+const MockSectionsJSON = `{
+  "_embedded": {
+    "clickindex:sections": [


The line above is still better than the 8 lines below.

Alexandre Abreu and others added 7 commits October 27, 2016 09:02

WIP: sections/search by sections/featured snaps store endpoints

98fe8d2

add sections completion

4f6b79c

Fix tests

41c3ead

merge master

f4ab82f

Add tests & fix incomplete test interfaces

de828be

Fix tests

13453a5

fix spread test for snap find

5cf0056

mvo5 reviewed Nov 17, 2016

View reviewed changes

AlexandreAbreu added 2 commits November 17, 2016 15:52

remove debug printf

b28fad9

Add sections tests; tweak find tests to handle sections

fc7b319

AlexandreAbreu changed the title ~~[WIP] Add new store endpoints & snap find support for sections handling~~ Add new store endpoints & snap find support for sections handling Nov 18, 2016

chipaca suggested changes Nov 21, 2016

View reviewed changes

mvo5 reviewed Nov 21, 2016

View reviewed changes

AlexandreAbreu added 4 commits November 21, 2016 10:23

Review tweaks

a17f6ec

Merge branch 'master' into store

810c6cc

Add retry

76019dc

go fmt nit

755e4bc

chipaca approved these changes Nov 21, 2016

View reviewed changes

niemeyer requested changes Nov 21, 2016

View reviewed changes

AlexandreAbreu and others added 3 commits November 22, 2016 16:22

tweaks: tests, cleanups

7fd5ff6

address review feedback, use s.retryRequest

f4acce2

Merge remote-tracking branch 'upstream/master' into HEAD

57324d5

mvo5 mentioned this pull request Nov 23, 2016

many: add support for sections #2333

Closed

mvo5 and others added 3 commits November 23, 2016 13:26

add spread test for snap find --section

0d935c6

set section explicitely in snap find (without further args)

ef811bf

Merge pull request #2 from mvo5/alex-store

b9413b9

small tweaks for the store branch

mvo5 added this to the 2.18 milestone Nov 24, 2016

Merge master; Fix conflicts

8cc6818

niemeyer approved these changes Nov 24, 2016

View reviewed changes

add test TestFindNothingMeansFeaturedSection

be1b745

mvo5 merged commit 4963489 into snapcore:master Nov 24, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add new store endpoints & snap find support for sections handling #2288

Add new store endpoints & snap find support for sections handling #2288

AlexandreAbreu commented Nov 17, 2016 •

edited

mvo5 Nov 17, 2016

chipaca left a comment

chipaca Nov 21, 2016

chipaca Nov 21, 2016

chipaca Nov 21, 2016

chipaca Nov 21, 2016

chipaca Nov 21, 2016

chipaca Nov 21, 2016

chipaca Nov 21, 2016

chipaca Nov 21, 2016

mvo5 left a comment

mvo5 Nov 21, 2016

AlexandreAbreu Nov 21, 2016

mvo5 Nov 21, 2016

mvo5 Nov 21, 2016

niemeyer left a comment

niemeyer Nov 21, 2016

mvo5 Nov 22, 2016

niemeyer Nov 21, 2016

niemeyer Nov 21, 2016

niemeyer Nov 21, 2016

niemeyer Nov 21, 2016

niemeyer Nov 21, 2016

niemeyer Nov 21, 2016

niemeyer Nov 21, 2016

niemeyer Nov 24, 2016

niemeyer Nov 21, 2016

niemeyer Nov 21, 2016

mvo5 commented Nov 23, 2016

mvo5 commented Nov 24, 2016

niemeyer left a comment

niemeyer Nov 24, 2016

niemeyer Nov 24, 2016

		@@ -153,7 +153,6 @@ The snap tool interacts with the snapd daemon to control the snappy software pla

		cmd, err := parser.AddCommand(c.name, c.shortHelp, strings.TrimSpace(c.longHelp), obj)
		if err != nil {

Add new store endpoints & snap find support for sections handling #2288

Add new store endpoints & snap find support for sections handling #2288

Conversation

AlexandreAbreu commented Nov 17, 2016 • edited

Choose a reason for hiding this comment

chipaca left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mvo5 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

niemeyer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mvo5 commented Nov 23, 2016

mvo5 commented Nov 24, 2016

niemeyer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AlexandreAbreu commented Nov 17, 2016 •

edited