Mb web interface #2327

tigranl · 2016-12-17T13:33:03Z

Added http server.

Freso

I like that the server only runs while importing, however, how will it behave when port 8000 is used by another process (e.g., if Picard is also running)?

Also, new code should generally be accompanied by tests too.

Freso · 2016-12-17T13:54:34Z

beets/ui/commands.py

@@ -545,7 +545,11 @@ def choose_candidate(candidates, singleton, rec, cur_artist=None,
            assert not singleton
            return importer.action.TRACKS
        elif sel == u'e':
-            return importer.action.MANUAL
+            ans = ui.input_("Use MB web interface?")


"MB" should probably be spelled out.

Freso · 2016-12-17T13:56:49Z

beets/util/mb_tagger_server.py

@@ -0,0 +1,22 @@
+import socket


You should add a copyright blurb, encoding info, and docstring to the file as well. (See other files.)

Freso · 2016-12-17T13:57:30Z

beets/util/mb_tagger_server.py

+import socket
+
+
+class Server:


Remember to include a docstring.

Freso · 2016-12-17T13:57:55Z

beets/util/mb_tagger_server.py

+        self.port = port
+        self.start = None
+
+    def start_server(self):


Docstring…

Freso · 2016-12-17T13:58:09Z

beets/util/mb_tagger_server.py

+        self.start.bind((self.host, self.port))
+        self.start.listen(1)
+
+    def listen(self):


… and docstring here too.

sampsyo · 2016-12-17T22:29:20Z

Hey! Thanks for forging right ahead on this. Can we chat a little about the design before you finish up?

In particular, I think this should be a plugin. Plugins can add options to the importer action prompt, so we can directly insert an option to launch the browser interface from there. This will avoid bothering people who don't need that option—for example, when running beets in an ssh instance on a remote server.

Do you have ideas about how you'll finish up the implementation? For example:

You'll want to let the user open their browser to the appropriate page. This can be done just by printing a URL or by hooking into the OS to actually open it. The latter would be more convenient.
To finish the server component, you can either do things manually or use a dependency such as Flask or the underlying Werkzeug library. The latter would probably make things simpler, but I wouldn't rule out the former if you want to avoid the dependency. Keeping things in a plugin will also avoid adding a dependency to beets core.

tigranl · 2016-12-17T22:42:11Z

Your last point is exactly what I thought. It's just a tiny http server, so there is no need to use Flask.

sampsyo · 2016-12-17T22:44:54Z

Oh yeah! I also meant to add that the edit plugin uses a custom importer prompt option—you might take a look there for inspiration.

tigranl · 2016-12-17T23:06:09Z

By the way, I was wondering why do you use Enum in the project? As far as I'm concerned, enum was added in the 3.4 version and it isn't supported in the 2.* version of python.

sampsyo · 2016-12-17T23:12:46Z

On 2.7, we use enum34, the backport available on PyPI.

This reverts commit 4ffc680.

tigranl · 2016-12-19T08:57:47Z

Hi, @sampsyo. How do you set up plugins? I have created beetsplug directory with __init__.py and web_tagger.py. And I have also appended directory of plugin to PYTHONPATH:
export PYTHONPATH=$PYTHONPATH:/home/dragonfly/beetsplug

But I am getting ImportError:
** error loading plugin web_tagger: Traceback (most recent call last): File "/usr/lib/python3.5/site-packages/beets-1.4.2-py3.5.egg/beets/plugins.py", line 254, in load_plugins namespace = __import__(modname, None, None) ImportError: No module named 'beetsplug.web_tagger'

sampsyo · 2016-12-19T15:18:21Z

Hi! The easiest way is probably just to use your existing checkout of the beets source (which I'm presuming you've "installed" with pip install -e .) and just create a new Python file in its beetsplug directory. This should get picked up automatically. That is, you can just create mbweb.py or whatever alongside edit.py and all the rest.

tigranl · 2016-12-19T15:35:46Z

That's what I did. I have a question about stages: can I just use threading instead of it? Or is it necessary to use due to beets architecture?

sampsyo · 2016-12-19T15:39:59Z

It's probably a better idea to use the special beets hooks for anything that can use them. But what were you imagining to do by using threading directly? I'd be happy to discuss in more detail if you have a specific proposal.

For this particular feature, however, I believe everything will need to take place on the "UI thread." That is, it should happen synchronously while the user is considering a single album—I can't see any obvious way to add parallelism here.

tigranl · 2016-12-19T16:07:13Z

I thought about running server in the separate thread, so the user can always choose another release. If the server is running in the UI thread you must rerun server every time you want to pick different track.

sampsyo · 2016-12-19T16:28:17Z

OK, I see. So the server would always be running in the background, but it would only be "meaningful" if there's some release currently being considered in the UI stage. Is that right?

That could make sense, although it could be a bit more complex than starting the server only when a choice is needed. This could be accomplished by starting the server in a background thread when the importer starts and shutting it down when it finishes.

tigranl · 2016-12-19T17:11:01Z

Yep, that's right. Do you think it would be an appropriate solution?

sampsyo · 2016-12-20T03:55:54Z

Sure! I'd advocate for whichever solution you find leads to the simplest code—the amount of functionality doesn't seem that different, overall.

tigranl · 2016-12-20T20:31:54Z

Some initial code: https://github.com/tigranl/beets-web-tagger
Sorry, didn't have time to code yesterday.

sampsyo · 2016-12-20T22:00:57Z

Oh, very nifty! Thanks for sending this along.

It looks like you have a little more coding to do (or, if I'm wrong, please feel free to summarize where you currently are). It also looks like the standard-library HTTPServer might be useful here?

tigranl · 2016-12-20T22:36:43Z

There still is some coding to do. I need to figure out how to pass user's 'Look up' choice to importer.action.MANUAL. Yeah, I thought about using HTTPServer, but I want to use sockets, because I have never actually tried it.

sampsyo · 2016-12-21T16:05:01Z

Cool!

For what it's worth, the function should probably return an autotag.AlbumMatch object, like this line that returns such an object during the normal tagging flow.

tigranl · 2016-12-24T12:09:47Z

Hi, @sampsyo. As you know, threads in python are implemented using GIL, so there is no way to run two threads simultaneously. But in my case I need to run server thread and beet main thread. So, I guess, the best idea would be to run server every time user wants to choose a track, or to use multiprocessing if user set up this option in the config file.

sampsyo · 2016-12-24T14:51:49Z

Hi! The GIL actually shouldn't be a problem here—it only becomes an issue if you want code to run faster by taking advantage of multiple cores. It won't stop you from running a web server on one thread and simultaneously doing other beets stuff in the other thread. The web server thread can be waiting for connections, for example, while the other thread makes progress.

tigranl · 2016-12-24T15:42:03Z

Should I create two separate threads with MBWeb class and ThreadedServer and then join() them?

sampsyo · 2016-12-25T01:31:27Z

Well, you don't actually need two threads—the main code is already running its own thread (implicitly). You just need to fork, and then eventually join, the second thread. Here's a tiny example, at the risk of being redundant:

import threading
import time


class MyThread(threading.Thread):
    def run(self):
        print('start thread')
        time.sleep(2)
        print('end thread')


def main():
    t = MyThread()
    print('before start')
    t.start()
    t.join()
    print('after end')


if __name__ == '__main__':
    main()

tigranl · 2016-12-26T18:16:26Z

Hi! Merry Christmas and a Happy New Year! 🎄
I am having trouble getting socket's listen run second time, can you give some advice on this?https://gist.github.com/tigranl/ccc7e26d23efa001ac25af125d099ee4

sampsyo · 2016-12-26T20:48:58Z

Happy holidays to you too!

Sure; I'd be happy to help! What's going wrong with that code? Is there anything I should try running locally, or maybe an error message I should look for?

tigranl · 2016-12-26T20:55:15Z

Try to run it locally.

sampsyo · 2016-12-26T20:58:27Z

OK—specifically, I should copy that gist into my beetsplug folder, enable the plugin, import something, and type l or whatever?

Any other context about what's going wrong?

tigranl · 2016-12-28T17:21:54Z

Yes, after adding socket send function it now works as expected. But I'm not sure about _get_plugin_choices argument, because it throws TypeError: _get_plugin_choices() missing 1 required positional argument: 'task'.
Also, i can't create PR with beetsplug directory, so I just added main script without __init__.py.

sampsyo · 2016-12-28T17:36:36Z

Great! A single-file plugin is just fine in this case.

Python is telling you this won't work:

TerminalImportSession._get_plugin_choices(task)

because that's an instance method, not a static method, so it can't be called on the class itself.

To make this work cleanly, we may want to extend the interface in the importer itself. It would be nice if the plugin didn't have to explicitly invoke the importer choice logic—if it could just return the ID in question and let the importer take care of the rest. That would avoid the whole problem of invoking stuff in TerminalImportSession altogether.

I'll look into this right now.

tigranl · 2016-12-28T18:32:35Z

Maybe we could use http://beets.readthedocs.io/en/v1.3.17/dev/plugins.html#extend-the-autotagger

sampsyo · 2016-12-28T19:25:58Z

OK, I've pushed a bit of refactoring that should make this easy! Now, the built-in option for searching by ID works using exactly the same functionality as a plugin. See this function here: https://github.com/beetbox/beets/blob/master/beets/ui/commands.py#L692-L707

That's a callback from a PromptChoice. As you can see, it just returns a Proposal value (also new, but it's just the return value from tag_album or tag_item) to expose the new possible matches to the user. Your plugin can use the same strategy: just call tag_abum (or tag_item in singleton mode) with the ID retrieved by the Web interface and return the Proposal. The normal workflow should take care of everything from there.

sampsyo

OK, looking good so far! This has made a lot of progress.

Before this can be merged, it will need more comments in the code and a documentation page. If you like, though, we can call this GCI task done.

sampsyo · 2016-12-29T19:01:11Z

beetsplug/web_tagger.py

+from beets import ui
+
+
+PORT = 8000  # Temporary for PR


This should probably be configurable.

sampsyo · 2016-12-29T19:01:39Z

beetsplug/web_tagger.py

+        else:
+            break
+    parsed = urlparse.urlparse(url)
+    return str(urlparse.parse_qs(parsed.query)['id'][0])


Comments for this funky parsing code would be really helpful.

I don't like this function very much. Any ideas how to rewrite? I am thinking of using regexp.

Sure, a regular expression would make sense. Can you give me an idea of what the code's supposed to do (it's a little hard to read without context)?

It retrieves Musicbrainz id from data received from the socket. Function makes the list of received bytes and picks up the first value (which is a GET method), then using for loop it catches URI, and finally parses id parameter with urlparse.

I hope my explanation makes things clearer.

Aha, got it! It seems like the right thing to do is to use a real HTTP parser then. Here's one way to do it, if you want to avoid using a full HTTPServer: http://stackoverflow.com/a/5964334/39182

sampsyo · 2016-12-29T19:02:16Z

beetsplug/web_tagger.py

+        self.register_listener('before_choose_candidate', self.prompt)
+        self.server = Server()
+        self.server.start()
+        self.server.join()


Any particular reason why the server is started and then joined at plugin startup time?

sampsyo · 2016-12-29T19:03:29Z

beetsplug/web_tagger.py

+    def choice(self, session, task):
+        artist = ui.input_('Artist:')
+        realise = ui.input_('Album:')
+        track = ui.input_('Track:')


We should extract these from the task instead of prompting for them.

(Also, realise -> release.)

sampsyo · 2016-12-29T19:04:35Z

beetsplug/web_tagger.py

+                     'realise': realise,
+                     }
+            url = 'http://musicbrainz.org/taglookup?{0}'.format(urlencode(query))
+            ui.print_("Choose your tracks and click 'tagger' button to add:")


This should probably end in a "." instead of a ":" because there's no prompt here.

sampsyo · 2016-12-29T19:05:12Z

beetsplug/web_tagger.py

+                _, _, proposal, _ = autotag.tag_album(task.items, search_ids=id_choice)
+                return proposal
+            else:
+                return autotag.tag_item(task.item, search_ids=id_choice)


Nice! I'm glad the new API works.

sampsyo · 2016-12-29T19:05:45Z

beetsplug/web_tagger.py

+        track = ui.input_('Track:')
+        if not (artist, realise, track):
+            ui.print_('Please, fill the search query')
+            return self.prompt


Any particular reason a method is returned here?

It's in case if user didn't provide information about track. Anyways, I'm gonna change it and use information from the task.

sampsyo · 2016-12-29T19:08:39Z

beetsplug/web_tagger.py

+                con.send(data)
+                if not data:
+                    break
+                return parse(data)


It looks like this Thread doesn't have a run method, which means this code is not actually running in a separate thread. That's fine—this version just blocks the main thread while waiting for a request from the browser—but we should probably just put this code into a plain function instead of a threading.Thread subclass.

sampsyo · 2016-12-29T19:10:18Z

Oh, also, for consistency with other plugins' names, I suggest the name mbweb instead of web_tagger. That also makes it clear that this is MusicBrainz-only.

sampsyo · 2016-12-30T15:42:25Z

Looking great! It also seems that Travis has a few style suggestions, if you're interested in applying those tweaks: https://travis-ci.org/beetbox/beets/jobs/187702547#L848-L854

tigranl · 2016-12-30T15:56:21Z

Ok. By the way, what should I do with init.py? Do I need to include it in the PR?

sampsyo · 2016-12-30T15:59:56Z

No, just the main code for the plugin here is fine.

tigranl · 2016-12-30T19:50:37Z

I left untouched two errors:
web_tagger.py:15:80: E501 line too long (85 > 79 characters) web_tagger.py:29:5: F811 redefinition of unused 'urlparse' from line 28
The first one is docstring.

sampsyo · 2016-12-30T21:09:30Z

Great! I took care of those two. (Meanwhile, I don't know what's going on with AppVeyor.)

The HTTP parser thing looks awesome, but the way. It's very short and easy to read.

I think this is close to ready to merge! All we really need now is documentation.

sampsyo · 2016-12-30T21:10:34Z

beetsplug/mbweb.py

+        self.host = '127.0.0.1'
+        self.port = PORT
+        try:  # Start TCP socket, catch soket.error
+            self.run_server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)


I might suggest the name sock or something for the socket, for clarity's sake.

sampsyo · 2016-12-30T21:10:48Z

beetsplug/mbweb.py

+    return parse_qs(parsed.query)['id'][0]
+
+
+class Server():


Should probably inherit from object.

sampsyo · 2016-12-30T21:12:07Z

beetsplug/mbweb.py

+                    .format(urlencode(query))
+                webbrowser.open(url)
+                id_choice = self.server.listen()
+                search_ids.append(id_choice)


Hmm; looks like this opens the browser for every track on the album? Maybe we just want to search by album?

But what if there are non-album tracks?

Hmm, I'm not sure I understand. Weird cases, like missing or extra tracks, would be handled by the ordinary import flow (exactly the same as if you had entered an ID manually).

That's what I was talking about. Okay then.

sampsyo · 2016-12-30T21:12:14Z

beetsplug/mbweb.py

+from beets.ui.commands import PromptChoice
+from beets import ui
+
+PORT = 8000


Move to configuration.

tigranl

Do I need to follow PEP8 on docstrings too? I saw a standard library file with a long docstring (>80) ee79e15

sampsyo · 2016-12-30T21:23:09Z

Yep! Sadly, the standard library isn't all that PEP8-clean. There are lots of old names likeThis, for example, even though PEP8 prescribes this_style.

jtpavlock · 2020-07-09T23:20:05Z

Is there still interest in getting this merged?

stale · 2020-11-18T00:50:55Z

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

tigranl added 2 commits December 17, 2016 16:26

Add http server and option to choose MB web interface

2bad974

Add http server and option to choose MB web interface

4ffc680

Freso requested changes Dec 17, 2016

View reviewed changes

Revert "Add http server and option to choose MB web interface"

889c7cb

This reverts commit 4ffc680.

Add arguments to 'choice' function

f3b30a9

tigranl force-pushed the MB_web_interface branch from f81066c to f3b30a9 Compare December 29, 2016 14:41

tigranl added 2 commits December 29, 2016 17:47

Add a 'proposal' value

53a6152

Add a 'proposal' value

47fb592

sampsyo requested changes Dec 29, 2016

View reviewed changes

sampsyo reviewed Dec 30, 2016

View reviewed changes

tigranl commented Dec 30, 2016

View reviewed changes

Add tests for plugin

6d4df1b

tigranl force-pushed the MB_web_interface branch from bb050e2 to 6d4df1b Compare January 3, 2017 14:18

Add documentation

1f67052

tigranl force-pushed the MB_web_interface branch 3 times, most recently from 7073508 to 1f67052 Compare February 2, 2017 21:54

tigranl added 3 commits February 3, 2017 00:58

Recover deleted commit

7198480

Add a newline

6417ad0

Rename

ac0082e

stale bot added the stale label Nov 18, 2020

stale bot closed this Nov 25, 2020

Mb web interface #2327

Mb web interface #2327

Conversation

tigranl commented Dec 17, 2016

Freso left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sampsyo commented Dec 17, 2016

tigranl commented Dec 17, 2016

sampsyo commented Dec 17, 2016

tigranl commented Dec 17, 2016

sampsyo commented Dec 17, 2016

tigranl commented Dec 19, 2016

sampsyo commented Dec 19, 2016

tigranl commented Dec 19, 2016

sampsyo commented Dec 19, 2016

tigranl commented Dec 19, 2016

sampsyo commented Dec 19, 2016

tigranl commented Dec 19, 2016 • edited

sampsyo commented Dec 20, 2016

tigranl commented Dec 20, 2016

sampsyo commented Dec 20, 2016

tigranl commented Dec 20, 2016

sampsyo commented Dec 21, 2016

tigranl commented Dec 24, 2016

sampsyo commented Dec 24, 2016

tigranl commented Dec 24, 2016

sampsyo commented Dec 25, 2016

tigranl commented Dec 26, 2016

sampsyo commented Dec 26, 2016

tigranl commented Dec 26, 2016

sampsyo commented Dec 26, 2016

tigranl commented Dec 28, 2016

sampsyo commented Dec 28, 2016

tigranl commented Dec 28, 2016

sampsyo commented Dec 28, 2016

sampsyo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tigranl Dec 29, 2016 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sampsyo commented Dec 29, 2016

sampsyo commented Dec 30, 2016

tigranl commented Dec 30, 2016

sampsyo commented Dec 30, 2016

tigranl commented Dec 30, 2016

sampsyo commented Dec 30, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tigranl left a comment • edited

Choose a reason for hiding this comment

sampsyo commented Dec 30, 2016

jtpavlock commented Jul 9, 2020

stale bot commented Nov 18, 2020

tigranl commented Dec 19, 2016 •

edited

tigranl Dec 29, 2016 •

edited

tigranl left a comment •

edited