ern: Add DDEX ERN Indexer #17

lmars · 2017-09-20T23:14:54Z

Summary:

Fix error reporting in ParseCwrFile
Switch to my fork of the CBOR library (github.com/lmars/cbor) to fix decoding tags (see Fix decoding tags into nil pointers whyrusleeping/cbor#8)
Add tests for the meta command
Add a DDEX ERN indexer

Signed-off-by: Lewis Marshall <lewis@lmars.net>

nolash · 2017-09-22T14:38:52Z

cmd/meta/main.go

@@ -253,14 +260,21 @@ func (m *Main) RunMusicBrainzConvert(args Args) error {
 		<-ch
 		log.Info("received signal, exiting...")


What does this signal trap actually do? I seems to perform no action. Should there be a rollback?

It traps the signal and then cancels the context which can then be handled gracefully by the converter.

Ah, the defer of course. I missed that.

nolash · 2017-09-22T14:42:06Z

cmd/meta/main.go

+
+	// run the converter in a goroutine
+	stream := make(chan *cid.Cid)
+	errC := make(chan error, 1)
 	go func() {


Why is this in a go-routine? Doesn't the ConvertArtists just run until completion and return the rows?

ConvertArtists sends CIDs to a channel as it converts them, when it returns there is no guarantee that the goroutine reading those CIDs has finished, so we run the conversion in a goroutine and only exit the program when the stream has been fully read below.

I've clarified in c8e62cc.

nolash · 2017-09-22T14:44:25Z

cmd/meta/main.go

+
+	// run the converter in a goroutine
+	stream := make(chan *cid.Cid)
+	errC := make(chan error, 1)
 	go func() {


same question as for musicbrainz above.

nolash · 2017-09-22T14:53:34Z

cmd/meta/main.go

+			cid, err := cid.Parse(s.Text())
+			if err != nil {
+				log.Error("error parsing cid", "value", s.Text(), "err", err)
+				return


Why return and not continue?

If there is a corrupt entry in the stream then we stop processing because it is an exceptional case (there should be no corrupt entries in a stream).

nolash · 2017-09-22T14:54:53Z

cmd/meta/main.go

+		defer cancel()
+		ch := make(chan os.Signal, 1)
+		signal.Notify(ch, syscall.SIGINT, syscall.SIGTERM)
+		<-ch


same question as for musicbrainz above

nolash · 2017-09-22T15:47:49Z

ern/index.go

+// Indexer is a META indexer which indexes a stream of META objects
+// representing DDEX ERNs into a SQLite3 database, getting the
+// associated META objects from a META store.
+type Indexer struct {


I expected to find an interface for Indexers somewhere, but can't. Are there Indexers totally disassociated?

Other than being META indexers, they are all slightly different at the moment, I don't think there is need for an interface.

they are all slightly different at the moment

My intuition tells me that sounds quite like an interface. We can wait and see, though, and see with time what the generics will look like.

nolash · 2017-09-22T15:56:20Z

ern/index.go

+		}
+		if err := i.indexProperty(ern.Cid(), id, indexFn); err != nil {
+			return err
+		}


What happens if some of the indexing functions already have been run when this fails and returns? Should the indexer entries in the database roll back?

The plan is that either indexing succeeds and the whole change to the DB is uploaded to Swarm, or indexing fails and nothing happens (so this whole process will eventually be wrapped in a "transaction" of sorts).

So the indexing (sqlite) db is kind of a private cache within the indexer node itself?

Yes, so I'm currently imagining doing:

$ swarm get index.db $ meta ern index index.db $ swarm put index.db $ geth --exec '<update-ens-record>'

With error checking in-between.

nolash · 2017-09-22T15:58:06Z

ern/index.go

+	for _, field := range []string{"ISRC", "CatalogNumber", "ProprietaryId"} {
+		if v, err := graph.Get("SoundRecordingId", field, "@value"); err == nil {
+			ids = append(ids, v.(string))
+		}


Please check the error

Errors are now being checked, see 25020a5.

nolash · 2017-09-22T15:58:27Z

ern/index.go

+	if v, err := graph.Get("ReferenceTitle", "TitleText", "@value"); err == nil {
+		title = v.(string)
+	}
+


Please check the error

nolash · 2017-09-22T16:16:01Z

ern/index_test.go

+	}
+
+	// check SoundRecording objects were indexed
+	for isrc, title := range map[string]string{


Judging from the code, an id for a SoundRecording can be ISRC,Catalognumber and ProprietaryId. Maybe test for all?

It also seems to only need either "SoundRecordingId" or ReferenceTitle. Maybe the test should also account for that?

Good spot, so I'm actually keen to run this on real ERNs before writing too much code around this (I actually don't think CatalogNumber and ProprietaryId are used so we may end up dropping them), so I'm going to leave the tests like this.

Signed-off-by: Lewis Marshall <lewis@lmars.net>

lmars added 6 commits September 21, 2017 00:13

cmd: Add tests for the 'meta' command

9457681

Signed-off-by: Lewis Marshall <lewis@lmars.net>

meta: Add IsPathNotFound

948323f

Signed-off-by: Lewis Marshall <lewis@lmars.net>

meta: Return a *cid.Cid from Graph.Get

613f5c6

Signed-off-by: Lewis Marshall <lewis@lmars.net>

meta: Fix test failure message

8fcd15d

Signed-off-by: Lewis Marshall <lewis@lmars.net>

vendor: Switch to github.com/lmars/cbor

d78cbfb

Signed-off-by: Lewis Marshall <lewis@lmars.net>

ern: Add DDEX ERN Indexer

d59ec73

Signed-off-by: Lewis Marshall <lewis@lmars.net>

nolash reviewed Sep 22, 2017

View reviewed changes

lmars added 3 commits September 22, 2017 19:40

cmd: Clarify why conversion happens in a goroutine

c8e62cc

Signed-off-by: Lewis Marshall <lewis@lmars.net>

migrate: Move SQLite3 migration code into migrate package

3e8b856

Signed-off-by: Lewis Marshall <lewis@lmars.net>

ern: Check errors in indexSoundRecording

25020a5

Signed-off-by: Lewis Marshall <lewis@lmars.net>

lmars mentioned this pull request Sep 22, 2017

Decide whether to alias package names #18

Open

nolash approved these changes Sep 23, 2017

View reviewed changes

lmars merged commit bb227a5 into master Sep 23, 2017

lmars deleted the ddex-ern-indexer branch September 23, 2017 21:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ern: Add DDEX ERN Indexer #17

ern: Add DDEX ERN Indexer #17

lmars commented Sep 20, 2017

nolash Sep 22, 2017

lmars Sep 22, 2017

nolash Sep 22, 2017

nolash Sep 22, 2017

lmars Sep 22, 2017

nolash Sep 22, 2017

nolash Sep 22, 2017

lmars Sep 22, 2017

nolash Sep 22, 2017

nolash Sep 22, 2017

lmars Sep 22, 2017

nolash Sep 22, 2017 •

edited

Loading

nolash Sep 22, 2017

lmars Sep 22, 2017

nolash Sep 22, 2017

lmars Sep 22, 2017

nolash Sep 22, 2017

lmars Sep 22, 2017

nolash Sep 22, 2017

nolash Sep 22, 2017

lmars Sep 22, 2017

		@@ -253,14 +260,21 @@ func (m *Main) RunMusicBrainzConvert(args Args) error {
		<-ch
		log.Info("received signal, exiting...")

ern: Add DDEX ERN Indexer #17

ern: Add DDEX ERN Indexer #17

Conversation

lmars commented Sep 20, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nolash Sep 22, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nolash Sep 22, 2017 •

edited

Loading