a few things / bugs I found while working extensively with huginn over the past few days

[level20peon]

I just want to compile a short list of things that I found to be wrong / inconsistent while working with huginn. I think they are just oversights, so I refrain from opening several issues for those:

1. The WebsiteAgent allows the url parameter to be an array, while it won't allow the url_from_event parameter to be one.
   One might ask oneself "why would a (one) url from an event need to be an array?". Well the thing is, if one don't pass a whole URL to url_from_event but only one parameter (like jsessionid) and want to append it to several different URLs in a WebsiteAgent, then this won't work.

2. I think nobody created more than just a few scenarios on one user account because there are no buttons being rendered for the page menu (page number, Next, Last) as soon as one creates more than 25 scenarios. Please see attachment as reference. Note that there are no errors logged so I think those images just aren't referenced to in the rendering code / css.

3. "back" always navigates to the overview instead of the specific view where one was before (i.e. all agents overview, all events overview, etc.). This goes for the "back" buttons found on several sub-pages as well as being the default target when editing an agent and then saving it. This is very annoying if one has got a few hundred agents and probably don't have the ones one just edited on the first page of that agents' list.
   I think that a better target would be the page where one actually clicked "edit" for an agent, or - in the case of "back" buttons - actually going back where one was before.

4. log rotation: my log file has amassed around 100MB of text during one day of being in production. This is quite a lot, if one considers that this will be 700MB in one week. Is there a log rotation implemented which will, at some point, either rotate or archive the log and start a new one (like apache does, for example) ?
   

[mikalauskas]

4. use logrotate. it's built-in tool. apache, nginx etc uses logrotate as well

[level20peon]

3. I am sorry to have overlooked your topic at Redirect back to scenario page #1002
   You already seem to have found this issue before me. I pulled the latest huginn version on september the first, so I didn't receive that fixed update at that point. I will test this as soon as I have time for an update.

4. thanks, I wasn't aware that this is not a built in apache function.

So that just leaves (1) url_from_event and (2) navigation buttons on scenarios open for now :)

[cantino]

I appreciate the feedback, thank you.

Would you be interested in trying to help fix any of these issues? Huginn is an open source project that we build in our free time, and all help is appreciated.

[level20peon]

As stated in my other thread, I don't speak ruby. I wouldn't want to add crappy code to a well maintained project, so all I can do in order to help is pinpoint what's broken (to a detailed level, that is), so here we go:

1. The url_from_event issue should be fixable by copying code sections over from the parser for url, which also is used in the WebsiteAgent. I just had a look at /app/models/agents/website_agent.rb which seems to contain the required sections but although I can code, I just cannot comprehend what the ruby code-workflow might be. So I hope someone with greater skills might take on this task.

2. The origin of that issue results from a mistake in the html generating code on the scenario pages:

This is the html output for the pagination on the agent view:

<ul class="pagination ">
    <li class="page active"><a href="/agents">1</a></li>
    <li class="page"><a href="/agents?page=2" rel="next">2</a></li>
    <li class="next_page"><a href="/agents?page=2" rel="next">Next ›</a></li>
    <li class="last next"><a href="/agents?page=2">Last »</a></li>
</ul>

This is the html output for the pagination on the scenario view:

<div class="pagination ">
    <ul>    
        <li class="page active"><a href="/scenarios">1</a></li>
        <li class="page"><a href="/scenarios?page=2" rel="next">2</a></li>
        <li class="next_page"><a href="/scenarios?page=2" rel="next">Next ›</a></li>
        <li class="last next"><a href="/scenarios?page=2">Last »</a></li>
    </ul>
</div>

So the culprit is that second ul not having the pagination class. I inserted that class into the html directly and it rendered correctly afterwards. (I assume that the whole <div class="pagination"> wrapper could be removed, too if we're already at it). Unfortunately I was not able to determine which code generates that specific part of the html output (some parts seem to be generated by ruby directly, using html templates, others seem to be injected via javascript, and even css seems to be used to append certain elements... you get why I am confused :) )

[cantino]

Thanks @level20peon, I'll take a look at these this weekend.

[cantino]

@level20peon, just to address your concern about not knowing Ruby, if you'd be interested in learning Ruby, I promise we won't let you "add crappy code to a well maintained project". I'm happy to give feedback via code review if you'd like to attempt any changes in Huginn. 😄 We'd love to have you involved in the code.

I've fixed the pagination issue in #1016 and allowed url_from_event to be an array in #1017. Thanks for the useful feedback!

[level20peon]

ok, pagination is working now, thank you!

However, the array doesn't quite seem to work:
http://pastebin.com/5y5AcNQr

This is what I tried ({{jsessionid}} is passed by another Agent):

{
  "expected_update_period_in_days": "14",
  "url_from_event": [
    "http://jobboerse.arbeitsagentur.de/vamJB/stellenangeboteFinden.html?execution=e1s1&_arbeitszeitHeimarbeit.wert.value=on&_arbeitszeitNachtarbeit.wert.value=on&_arbeitszeitSchicht.wert.value=on&_arbeitszeitTeilzeitAbend.wert.value=on&_arbeitszeitTeilzeitFlexibel.wert.value=on&_arbeitszeitTeilzeitNachmittag.wert.value=on&_arbeitszeitTeilzeitSchicht.wert.value=on&_arbeitszeitTeilzeitVormittag.wert.value=on&_arbeitszeitVollzeit.wert.value=on&_arbeitszeitWochenende.wert.value=on&_behinderung.wert.value=on&_eventId_stellenangeboteSuchen%3Dnull%26request.parameter.jb.ankerhash.link.name%3Da358706291=Stellenangebote%20suchen&_keineAehnlicheBerufe.wert.value=on&allgemeineSuchkriterien.eingeklappt=false&arbeitsort.ort.wert=&arbeitsort.plz.wert=50677&arbeitsort.region.wert.wert=10&arbeitszeitVollzeit.wert.value=true&befristung.wert.wert=&beginnDerTatigkeit.wert=&berufe.wert=&betriebsgrosse.wert.wert=&fuhrungsverantwortung.wert.wert=&geringfugigeBeschaftigungMiniJob.wert.wert=2&jb.flow.event.hauptfunktion=_eventId_stellenangeboteSuchen&keineAehnlicheBerufe.wert.value=true&nurStellenMitFolgendenBegriffen.wert=betriebswirtschaftslehre&nurStellenOhneFolgendeBegriffe.wert=&stellenangeboteDerLetzten.wert.wert=2&suchbegriffebeziehung.wert.wert=2&umkreis.wert=50&weitereSuchkriterien.eingeklappt=false&o=1&jsessionid={{jsessionid}}",
    "http://jobboerse.arbeitsagentur.de/vamJB/stellenangeboteFinden.html?execution=e1s1&_arbeitszeitHeimarbeit.wert.value=on&_arbeitszeitNachtarbeit.wert.value=on&_arbeitszeitSchicht.wert.value=on&_arbeitszeitTeilzeitAbend.wert.value=on&_arbeitszeitTeilzeitFlexibel.wert.value=on&_arbeitszeitTeilzeitNachmittag.wert.value=on&_arbeitszeitTeilzeitSchicht.wert.value=on&_arbeitszeitTeilzeitVormittag.wert.value=on&_arbeitszeitVollzeit.wert.value=on&_arbeitszeitWochenende.wert.value=on&_behinderung.wert.value=on&_eventId_stellenangeboteSuchen%3Dnull%26request.parameter.jb.ankerhash.link.name%3Da358706291=Stellenangebote%20suchen&_keineAehnlicheBerufe.wert.value=on&allgemeineSuchkriterien.eingeklappt=false&arbeitsort.ort.wert=&arbeitsort.plz.wert=50677&arbeitsort.region.wert.wert=10&arbeitszeitVollzeit.wert.value=true&befristung.wert.wert=&beginnDerTatigkeit.wert=&berufe.wert=&betriebsgrosse.wert.wert=&fuhrungsverantwortung.wert.wert=&geringfugigeBeschaftigungMiniJob.wert.wert=2&jb.flow.event.hauptfunktion=_eventId_stellenangeboteSuchen&keineAehnlicheBerufe.wert.value=true&nurStellenMitFolgendenBegriffen.wert=business%20intelligence&nurStellenOhneFolgendeBegriffe.wert=&stellenangeboteDerLetzten.wert.wert=2&suchbegriffebeziehung.wert.wert=2&umkreis.wert=50&weitereSuchkriterien.eingeklappt=false&o=1&jsessionid={{jsessionid}}"
  ],
  "type": "html",
  "mode": "on_change",
  "extract": {
    "content": {
      "css": "table#tabellenanfang tbody tr",
      "value": "./node()"
    }
  }
}

[cantino]

Hey @level20peon, I don't think you pulled in my allow_url_from_event_to_be_array branch from #1017? When using that, I get further, but then run into the error in #1018 that @knu is looking at.

[level20peon]

Oh I'm sorry, I was unaware that I have to check out another branch than the master. Apparently I still have a lot to learn regarding git.
However, pulling that branch enabled me to use that array of urls - thank you!

[cantino]

Great! I just merged that branch into master, so you can switch back now. Thanks for testing it!

[cantino]

Should I close this Issue now?

[level20peon]

You've done enough for today ;)