Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problem: Watched folders for automatic import/filesystem based import not included in analyzer. #70

Open
Robbt opened this issue Mar 12, 2017 · 23 comments

Comments

@Robbt
Copy link
Member

commented Mar 12, 2017

I believe this functionality is missing from analyzer but the ability to bulk import tracks via FTP or the filesystem is pretty useful especially for people starting to use LibreTime from a different automation system. This was called the watched folders feature.

@hairmare

This comment has been minimized.

Copy link
Member

commented Mar 13, 2017

There is the ftp upload hook that might replace parts of the watched folder feature and I also found airtime-import in utils which seems to be somewhat related.

@hairmare

This comment has been minimized.

Copy link
Member

commented Mar 13, 2017

I have the UI changes prepared on master...radiorabe:feature/restore-media-folders-ui.

The calls to media-monitor that need to get changed to analyzer are on airtime_mvc/application/models/MusicDir.php#L296, #L436 and airtime_mvc/application/controllers/PreferenceController.php#L432

I started looking at media-monitor but didn't get very far, may re-implementing this in analyzer greenfield style makes more sense. Basically analyzer needs to handle the new_watch, remove_watch and rescan_watch amqp messages.

@samcarmex

This comment has been minimized.

Copy link

commented Nov 1, 2017

Just wondering if there's any update on this? Monitoring a network mounted folder is now impossible without the airtime-import function working. We store all our audio content on a separate server (same local network though) and simply 'import' the files into Airtime without having to copy them to a second folder. This can't be done with Libretime (AFAIK).

Are there any plans to implement a true watched folder functionality or will that not be a priority?

@Robbt

This comment has been minimized.

Copy link
Member Author

commented Nov 2, 2017

Work has been done under #111 but I'm not sure what state it is in. Either the original contributor @HaJoHe will need to finish the work or someone else will need to do whatever is left that needs to be done to bring this functionality back. I wouldn't say it's not a priority but it hasn't been on the top of my todo list at the moment.

@HaJoHe

This comment has been minimized.

Copy link
Contributor

commented Nov 8, 2017

Hi,
all works fine for me. So, if there are any modifications needed, I'll try to do it.

@Robbt

This comment has been minimized.

Copy link
Member Author

commented Dec 1, 2017

Ok, I had an installation issue w/ your PR. I was getting permission errors writing to the egg file. I'm trying it again from a fresh vagrant box and see if I can get it to work. It has been a regularly requested feature and we should commit the PR if it works out of the box.

If I remember correctly the only major objection is that you are writing directly to the database vs. going through the API. I'm not sure if this will cause issues down the line but I'm not sure any hooks happen that alter the files on their way through the system anyways.

@samcarmex

This comment has been minimized.

Copy link

commented Jan 19, 2018

Just wondering if any further work has been done on this issue? Watched folder functionality still doesn't exist AFAIK and is the single reason I'm still clutching onto my outdated Airtime install. Really wanting to upgrade to LibreTime but doesn't seem like it's there yet?

@Robbt

This comment has been minimized.

Copy link
Member Author

commented Jan 19, 2018

I started to look into it and test it. It appears at first glance that there is an issue with the python magic module since she switched after this was coded. I agree that this is probably the highest priority for the project at the moment as it makes adopting LibreTime rather time consuming. If I have the time I plan on doing whatever I can to get this finished.

@Robbt

This comment has been minimized.

Copy link
Member Author

commented Oct 12, 2018

I did some work with watched folders but when investigating the way airtime-analyzer imports code I found it required a lot of hacking the PHP files to basically pass a new parameter that this file was a copy and already exists in place. It also didn't ultimately work when I tried to playback the files and so I was heading more in the direction of automatic filesystem import where files were deleted after being imported.

So there is a question as to how important it is for a watched folder where files exist outside of LibreTime and are synced to the database.

There has been some discussion about this on the forum.

@hairmare

This comment has been minimized.

Copy link
Member

commented Oct 12, 2018

IMO this is one the larger blockers since some stations have large audio archives (>1TiB) that would get duplicated if we have users upload them to LibreTime.

@Robbt

This comment has been minimized.

Copy link
Member Author

commented Oct 12, 2018

Yeah I think that we need to really think about how he we do it if we want to keep it as it adds a lot of potential for failure if the files are remotely hosted.

Also it may make sense to add something to the UI to indicate a file is remotely hosted or part of a watched folder. The other problem that maintaining watched folders could create is it doesn't add files that are uploaded to watched directory and so now the station will have 2 repositories of tracks to maintain.

@hairmare

This comment has been minimized.

Copy link
Member

commented Oct 12, 2018

If I understood the old code correctly syncing the watched folders to the database was done by cron regularly, maybe looking into inotify and having a libretime-watch service could also make sense.

@Robbt

This comment has been minimized.

Copy link
Member Author

commented Oct 12, 2018

Yeah I was using pynotify which uses inotify for the LibreTime-import service I wrote but being inotify based it won't support remote filesystems such as sshfs. The biggest challenge is figuring out the minimalist changes to support imports that don't change the filepath while using the rest media API.

@hairmare

This comment has been minimized.

Copy link
Member

commented Oct 13, 2018

We certainly need to extend the REST API to be able to do this properly. I looked into this ages ago and it's clearly a cross cutting concern that needs lots of work.

@Robbt

This comment has been minimized.

Copy link
Member Author

commented Oct 14, 2018

Yeah, so this is why I'm not sure we want to include "watched folders" before we start the rewrite of the backend. I tried to cludge my way through modifying @HaJoHe PR in a separate branch but I found the process of adding parameters on multiple PHP functions that were chained together to simply say "don't copy this file but use the filesystem path being passed" to be problematic. I could go back and try it again or else write a whole separate API call for "pre-existing files" that attempts to simplify it. Or we could just do something with a filesystem import service that allows people to add tracks via the filesystem and moves them into LibreTime. Either way #508 should be fixed so that the problem files don't clog the import directory.

My personal take on this is if people have a well organized library of audio that they want to maintain they should do this on a separate computer from their LibreTime server and copy the files into LibreTime to be imported and stored separately on the LT file system. We probably need to add documentation for how to do this via FTP or SSH or via flash drive what not.

The only time there would be duplication of files is if they installed LT on a AT system where they had maintained a separate directory that they watched and then they install LT and the "watched folders" tracks get into an unknown state. Actually the question of what happens to "watched folders" and the tracks on a upgraded LT is a good question and probably needs to be addressed especially if we decide to not support this feature.

But I think for a lot of people being able to upload tracks via the file system will solve the problem of requiring them to manually upload a bunch of tracks through the web interface. Some kind of web-export/mass download would also be useful because I know people using airtime.pro have no way of getting their data out and transitioning to LT other than manually downloading everything.

@Robbt

This comment was marked as off-topic.

Copy link
Member Author

commented Oct 14, 2018

Well it would be helpful if you didn't characterize a code review/critique as a personal attack Everyone communicates differently and it is important for the sanity of the project to assume good will when someone is critical or questioning of a contribution. In my opinion we don't have a situation where people are being personally attacked or insulted so it'd be great if you could try to not take things personally.

I think the core disagreement that has happened and you reference is more of a contradiction between two different coding philosophies, the quick hack vs. the optimal solution. In my opinion as a project we are striving for the optimal while wading through the muck of technical debt that makes up our codebase. I think we should provide some kind of outlet for the kind of one-off scripts and alternative RSS importers that you have developed. If you want to test them out in your fork and get them working then perhaps we can bring them back later.

But I'm also of the mindset that we should try to do the best we can and verify that our code is correct before we commit it. In an ideal world every piece of code contributed would have corresponding documentation and tests to verify its validity but we have a very bloated codebase already that is lacking in these regards.

On the other hand a lot of your scripts might be helpful if added to something like a utils/contrib folder with accompanying documentation so people could use them as they needed.

I wrote a python service #514 for the import script which probably does essentially the same thing as your script but just conforms more to our current service orientated paradigm and once I add documentation and we decide upon whether we want to support "watched folders" or copying existing files vs. just files that are copied into this directory then we can merge it.

@Robbt

This comment was marked as off-topic.

Copy link
Member Author

commented Oct 14, 2018

@JohnnyC1951 I'm sorry you feel that way. I think you are referring to the script posted here - #181 - and I don't see anything overtly personal or attacking in @hairmare's response:

Your description sounds like import_a_track.sh does exactly what ftp-upload-hook.sh should do (and calling it for multiple files is a find one-liner in my book). If there is an issue with the upload hook, please describe it properly so we can get it fixed.

I can see how that response could be seen as dismissive of the contribution but I don't think it should be taken as an personal attack. I found the script posted there to be useful but like I mentioned above figuring out a way to include scripts and hacks that aren't part of the web UI or core of the project. But yeah, I still stand behind the point I made above. It is very easy to perceive malice when reading the written word. Like I said above we should try to be respectful of each other and work together. The best thing this project has going for it is community and constantly jumping in and out and getting upset is not helpful towards the vibe.

I know the AT forum had some contributors who were just venomous at times and slung insults and bizarre metaphors in a negative fashion but I don't see this type of negativity being directed towards you on here. If others see things differently they can chime in. We should be trying to build a cooperative project and like I said above I think the scripts that you use are useful but should be added to a part of the project where people can find them and use them. We don't want the github issue queue to become a repository for hacks etc, those could be posted on the forum but I'd prefer the ones that are useful to be added to utils/contrib or wherever with at least minimal text documentation. If you want to contribute them via PR.

@Robbt

This comment was marked as off-topic.

Copy link
Member Author

commented Oct 14, 2018

Which is exactly where I see the problem occurring. I did find ftp-upload-hook.sh to work when I finally tried to figure out what was happening. @ghost got offended and immediately projected an antagonistic attitude on what I think was a reasonable question. Honestly we all have different personalities here and we might rub each other the wrong way at times or perceive hostility that wasn't intended. It isn't helpful if you are going to now go around being antagonistic and passive aggressive because you feel like your contributions weren't appreciated as much as they should have been.

Can you try to let bygones be bygones and not bring up perceived snubs of the past so that we can all try to foster a cooperative atmosphere and not one rife with tension. I am not trying to dismiss your feelings as invalid, I understand that everyone perceives reality in a different way but it is important for the project as a whole that we try and treat each other with respect even if we have different opinions about code. This is essential if this project is going to survive and reach its full potential.

@JohnnyC1951

This comment was marked as off-topic.

Copy link

commented Oct 15, 2018

Robby is in Canada reprogramming the Arm Boxes. I agreed with him.
If you look over my stuff from today... I am not sure it was me who actually blew. I fed a load of useful stuff in and removed some that wasn't appreciated.

We differ in that I consider LT as a tool to do a job. If it doesn't fit my needs, fix it and share the fixes.
Some seem to think it should be some perfect application, ticking every standards box and it doesn’t matter how late it is. There we differ.

If I had waited until Logitech & Mustek were perfect... :\

@JohnnyC1951

This comment was marked as off-topic.

Copy link

commented Oct 15, 2018

'This is essential if this project is going to survive and reach its full potential.'
No you will do fine without me, if even later in delivery.
Are we all done?

@JohnnyC1951

This comment was marked as off-topic.

Copy link

commented Oct 15, 2018

OK we are all done. Notifications off.

@Robbt

This comment has been minimized.

Copy link
Member Author

commented Jan 18, 2019

So this has been targeted for the beta but in reality we need to make a decision as to how we plan on implementing it and whether we want to rewrite analyzer to accommodate watched folders or just say that we no longer support this and provide a CLI tool for people to import files from their filesystem.

@frecuencialibre

This comment has been minimized.

Copy link
Contributor

commented Jan 18, 2019

i think this discussion is now in #514. my memory is that following my testing on a full library we concluded that pynotify isn't the best choice. my vote is for a CLI tool. i also noticed recently that the script that @hairmare mentions is in the documentation, so that should be updated if need be.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.