New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

download logs on demand #762

Closed
schakrava opened this Issue Jul 28, 2015 · 27 comments

Comments

Projects
None yet
5 participants
@schakrava
Member

schakrava commented Jul 28, 2015

Provide a feature to download latest log.tgz file from the web-ui.

@schakrava schakrava added this to the Ubehebe milestone Jul 28, 2015

@schakrava schakrava modified the milestones: Yosemite, Ubehebe Oct 4, 2015

@MFlyer

This comment has been minimized.

Member

MFlyer commented Apr 15, 2016

Hi @schakrava, can i work on this? Think it can be useful

I can see the assignee, but think it's not active, am i wrong?

Flyer

@schakrava

This comment has been minimized.

Member

schakrava commented Apr 15, 2016

You are more than welcome to, @MFlyer. this is indeed an important feature and it's safe to assume that @gkadillak is not actively working on it.

MFlyer added a commit to MFlyer/rockstor-core that referenced this issue Apr 16, 2016

MFlyer added a commit to MFlyer/rockstor-core that referenced this issue Apr 18, 2016

MFlyer added a commit to MFlyer/rockstor-core that referenced this issue Apr 19, 2016

MFlyer added a commit to MFlyer/rockstor-core that referenced this issue Apr 19, 2016

MFlyer added a commit to MFlyer/rockstor-core that referenced this issue Apr 19, 2016

@MFlyer

This comment has been minimized.

Member

MFlyer commented Apr 19, 2016

Hi All @schakrava @phillxnet @priyaganti , here is a first preview of what planned for logs access via Rockstor Web UI Log on demand preview

I thought having it with just a link to download rockstor logs would have been like a clone of actual error system, so i've decided to let user choose which logs to donwload on demand, plus adding a "live log reader" - to be added modal window for loading files cat/tail

Any suggestion about layout & logs to add? 😊 Actually I've got template and view good skeleton and adding on them now it's easy, backend part is the missing one (I'm used to start always from frontend beacuse it's my "babau" and my frontend coding is a little rusty comparing to backend)

@phillxnet

This comment has been minimized.

Member

phillxnet commented Apr 19, 2016

@MFlyer This is great, well done. A small suggestion to assist with understanding what's what though:
you could put (kernel) in brackets after Dmesg akin to what you have done to explain smbd/nmbd. Same could be done for the Yum entry ie (packaging) or (updates), also for Nginx ie (WebUI).
Also the addition of a general system / systemd log akin to journalctrl would be good, currently just nicer time stamps but going forward there is a greater split between these. There might also be the option to employ systemd's log filtering capabilities (extensive), but that's most likely not a first throw feature in my opinion.

Not looking as closely as your self so take all of the above with a grain of salt.

This is going to be a great addition.

MFlyer added a commit to MFlyer/rockstor-core that referenced this issue Apr 19, 2016

@MFlyer

This comment has been minimized.

Member

MFlyer commented Apr 19, 2016

Hi @phillxnet, just added your suggestions (brackets adds).

Talking about systemd-systemctl /journalctl : totally agree with you, but got some doubts for outputs from systemctl. Ok on journalctl, it's easy (long log, but like others)

Ex. this is a systemctl list-units -t service just to limit items, but it still outputs a long log with long items too (fsck)

system
We risk to have few or too much infos, depending on how it get implemented 😕 - @schakrava looking the image, why are we running wpa_supplicant?!?

We could get a select with services like options (green bg if loaded and active, red if failing) and fetch service details on select change, but probably that would overload users with datas

@MFlyer

This comment has been minimized.

Member

MFlyer commented Apr 19, 2016

Hi all, maybe I'm getting wrong or just don't see an error, but got some problems adding new url to urls.py

added (r'^api/systemlog/', include('storageadmin.urls.logs')), to urls.py
added logs.py under storageadmin/urls with just one line pattern for testing import

from django.conf.urls import patterns, url
from storageadmin.views import (LogsManagerView)

urlpatterns = patterns(
'',
url(r'^$', LogsManagerView.as_view(),),
)

added logs_manager.py under views folder and LogsmanagerView class it's a full clone of another existing Rockstor module, so it can't be broken, but after build of py i'm getting this on browser:
ImportError at /
cannot import name LogsManagerView
Request Method: GET
Request URL: https://rockstone/
Django Version: 1.6.11
Exception Type: ImportError
Exception Value:
cannot import name LogsManagerView
Exception Location: /opt/build/src/rockstor/storageadmin/urls/logs.py in <module>, line 20

Meanwhile doing my job so maybe there's an error just in front of me and i can't see it 😕

@gkadillak

This comment has been minimized.

Contributor

gkadillak commented Apr 19, 2016

There's a broken path at the top of your file in the imports in the logs.py file.

@MFlyer

This comment has been minimized.

Member

MFlyer commented Apr 20, 2016

Don't care about my last post, I was using my class (correctly called via urls, include, etc etc) but without importing it to views/__init__.py 😭

Ufff!!
Flyer

MFlyer added a commit to MFlyer/rockstor-core that referenced this issue Apr 21, 2016

MFlyer added a commit to MFlyer/rockstor-core that referenced this issue Apr 22, 2016

rockstor#762 Modified template and view for logs manager - Added moda…
…l for live log reader (template) inside WebUI + mods and beautify to view (ajax call plain text + render in modal)

MFlyer added a commit to MFlyer/rockstor-core that referenced this issue Apr 22, 2016

@MFlyer

This comment has been minimized.

Member

MFlyer commented Apr 22, 2016

Hi @schakrava @phillxnet @priyaganti , let's see this (first test on "inline log reading", going to add overflow etc etc etc)

https://youtu.be/qds2xtIN7C0

To @schakrava : found my way to hack django MVC usual structure, without @api_view and with a class view acting how i want (performing simple api tasks -> exec command + response plain text without care for json and letting user to have an easy readable log )

MFlyer added a commit to MFlyer/rockstor-core that referenced this issue Apr 23, 2016

MFlyer added a commit to MFlyer/rockstor-core that referenced this issue Apr 23, 2016

rockstor#762 Removed logs.js testing func - on previous commit: added…
… modal fix height to 300px to allow overflow-y; added log name and reader type in modal header; cat changed to cat -n allowing easy referencing lines

MFlyer added a commit to MFlyer/rockstor-core that referenced this issue Apr 24, 2016

MFlyer added a commit to MFlyer/rockstor-core that referenced this issue Apr 24, 2016

MFlyer added a commit to MFlyer/rockstor-core that referenced this issue Apr 24, 2016

MFlyer added a commit to MFlyer/rockstor-core that referenced this issue Apr 24, 2016

MFlyer added a commit to MFlyer/rockstor-core that referenced this issue May 10, 2016

MFlyer added a commit to MFlyer/rockstor-core that referenced this issue May 10, 2016

MFlyer added a commit to MFlyer/rockstor-core that referenced this issue May 10, 2016

MFlyer added a commit to MFlyer/rockstor-core that referenced this issue May 10, 2016

rockstor#762 Added functionality to risize Live Log Reader modal for …
…a better view - next step add func to resize font-size

MFlyer added a commit to MFlyer/rockstor-core that referenced this issue May 10, 2016

MFlyer added a commit to MFlyer/rockstor-core that referenced this issue May 10, 2016

@MFlyer

This comment has been minimized.

Member

MFlyer commented May 10, 2016

Hi all,
while working on rotating logs had fun adding some minor features

  • More logs available (obviously open to suggestions for more/different logs!)
  • Modal window enlarge/reduce button
  • Code text resize buttons (we care about users eyes health eheh)

Here a small video, showing logs .tgz direct download from log reader too :)

https://youtu.be/vwcJvRrFirY

@maxhq

This comment has been minimized.

Contributor

maxhq commented May 12, 2016

@MFlyer This is a really great module! I like the page layout and the functions.

There is only one thing that's unclear to me: does the "Live Log Reader" update when there are new log messages coming in while it's open? I.e. is it a tail -n 30 or a tail -f -n 30?

@MFlyer

This comment has been minimized.

Member

MFlyer commented May 12, 2016

Hi @maxhq, great question : actually tail -n 30 means what you read and not tail -f

In the early development I thought about having tail -f too, but I'm afraid about possible deadlocks

For a better understanding (@schakrava can confirm): Dashboard and widgets, like sysinfo updates, work on gevent.socketio with a nice trick (on connection some switches, one for every widget, turn to true and data is pushed to client forever till disconnection - switches auto turn to false as disconnection detected, builtin gevent.socketio method)
Logs reading is a little different: every action start on an on_*, where * is reading logs or building logs archive, so we would have to had a..

...Ok, while reading what i was writing i found my way 😁 : we can do that, just paying attention. I'll have to add a "special" log reader working with a safety switch too (one more method to stop tail -f on modal closed + switch turned to false on disconnection to prevent process running after browser closing!!!)
@phillxnet and @schakrava , what do you think about this?

@phillxnet

This comment has been minimized.

Member

phillxnet commented May 12, 2016

@MFlyer I'm afraid I am not up enough on these web technologies to comment on the possible ramifications. I agree that a live feed would be dandy but this is already a fairly major patch so there may be some sense in moving additional functionality into another issue/pr so that we might get what has been tested so far into play. Also there are re-factoring changes ongoing in the UI code that may make additional delays impact disproportionately on merge ease. Though I haven't really looked at this element re this pr. Just noting in case.

Although @maxhq idea of a live feed has great utility, so does what you have done so far. Also I would be concerned that the testing you have done to date may be negated by feature creep this late on. However please qualify my comments here on me not being very up on the web tec side so just a general concern really.

@phillxnet

This comment has been minimized.

Member

phillxnet commented May 12, 2016

@MFlyer Oh and what @maxhq said: this is looking really great.

MFlyer added a commit to MFlyer/rockstor-core that referenced this issue May 13, 2016

MFlyer added a commit to MFlyer/rockstor-core that referenced this issue May 13, 2016

rockstor#762 Backend mods for realtime reader tail -f. Splitted logre…
…ader in static and live reader + back to spawning to handle tail -f kill on modal closed. To @schakrava: already tested and nicely working, actually code is a little messy/redundant - going to revision
@MFlyer

This comment has been minimized.

Member

MFlyer commented May 13, 2016

Hi @maxhq , @phillxnet and @schakrava , tail -f added on d3e5df3 + 42f7421 😃

Code now need revision (optimization for redundancies) + rotating log handling

Flyer/Mirko

@schakrava schakrava modified the milestones: Looney Bean, Yosemite May 13, 2016

MFlyer added a commit to MFlyer/rockstor-core that referenced this issue May 16, 2016

MFlyer added a commit to MFlyer/rockstor-core that referenced this issue May 16, 2016

MFlyer added a commit to MFlyer/rockstor-core that referenced this issue May 16, 2016

rockstor#762 Rotating logs 2/2 - Frontend handles rotating logs both …
…for reading and downloading- Coding ended, needed only revision
@MFlyer

This comment has been minimized.

Member

MFlyer commented May 16, 2016

Hi all, finally Logs Manager coding is ended with full support for rotating logs! 😃

To @schakrava : Going to revision code, will tell you asa PR is ready for merging/your revision 😉

Flyer/Mirko

MFlyer added a commit to MFlyer/rockstor-core that referenced this issue May 17, 2016

MFlyer added a commit to MFlyer/rockstor-core that referenced this issue May 17, 2016

MFlyer added a commit to MFlyer/rockstor-core that referenced this issue May 17, 2016

rockstor#762 Code revision 1/2 - Frontend last checks plus added inte…
…lligent log switching for archive builder - rotated logs always on bottom, current logs always on top

MFlyer added a commit to MFlyer/rockstor-core that referenced this issue May 18, 2016

rockstor#762 Code revision 2/2 - This commit close backend revision, …
…added spawning to all funcs and tested, plus removed some redundant code - Ready for PR merge via @schakrava final code revision
@MFlyer

This comment has been minimized.

Member

MFlyer commented May 18, 2016

Code revision finished, updates on PR #1312

MFlyer added a commit to MFlyer/rockstor-core that referenced this issue May 19, 2016

rockstor#762 Firefox broken websockets on link clicking+new connectio…
…n established and rotated logs repeating[solved] : On Logs downloader added download empty attribute to download link, added hidden download link with download attribute empty on log reader modal too, so we trigger it on current log download request

MFlyer added a commit to MFlyer/rockstor-core that referenced this issue May 19, 2016

@schakrava

This comment has been minimized.

Member

schakrava commented May 20, 2016

Fixed by #1312

@schakrava schakrava closed this May 20, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment