Skip to content

Commit

Permalink
gateone.py: Removed OpenLogHandler since it isn't being used anymore.
Browse files Browse the repository at this point in the history 
gateone.py:  Fixed the typo in the new default web server logs location.  This should resolve #81
gateone.py:  A few more changes here and there to use os.path.join() instead of assuming the separator is "/".
gateone.py:  The container and prefix are now stored as instance variables of TerminalWebSocket (so they could be used by the Logging plugin).
termio.py:  The log format now includes the number of rows/cols in the metadata.
termio.py:  Made some minor adjustments to the blocked_io_handler (and related checks/functions) to make that whole watchdog-style mechanism more efficient.  Seems to kick in a lot faster now when someone runs a command like "yes".
gateone.js:  Minor change to loadWebWorkerAction() to prevent an exception that can occur when closing terminals (related to the now-gone terminal's title).
gateone.js:  Added a new function to GateOne.Utils:  getOffset().  Does what you'd think.
gateone.js:  The container and prefix are now sent along with the authentication JSON message.
gateone.js:  GateOne.Terminal has been placed inside its own sandbox in order to avoid a circular reference with the GateOne object when executing terminal refreshes.  NOTE: According to the debugger it seems to have worked (no more circular GateOne object references).  Also according to the debugger it seems to have made absolutely no difference in memory consumption.
gateone.js:  Changed how terminal updates are processed slightly in order to avoid creating circular references that could prevent proper JavaScript garbage collection.  In theory, these changes are for the better but I didn't notice much of a change in terms of memory utilization.  Also, Chrome 16 has a memory leak of some sort (with Gate One) that I couldn't seem to work around (which is what I was trying to do with these changes).  Fortunately this problem appears to be fixed in Chrome 17 and you can always reclaim memory by simply reloading Gate One in your browser window.
gateone.js:  Added the ability to disable the bell sound (a visual indication will still appear).  This should resolve #80
logviewer.py:  Moved some of the functions from logging_plugin.py into logviewer.py since it makes more sense to have them there.
terminal.py:  Minor adjustment to the title sequence and special optional escape sequence regular expressions so they won't get confused and have you stuck in a mode where the terminal emulator thinks you're endlessly entering a title or a special optional escape sequence.
utils.py:  ^J and ^M have been commented out of REPLACEMENT_DICT because they were causing issues when playing back logs (which sort of defeated the purpose of that dict in the first place).
Bookmarks Plugin:  Minor change to bookmarks.js to prevent a possible (transient) exception when the iconQueue isn't present while loading.  Also cleaned up a few variables that weren't being used.
Playback Plugin:  Playback of session recordings is now working (again).  I also fixed the logic that will re-size the terminal to fit within the current window.
Logging Plugin:  BIG CHANGE/NEW:  Viewing and playback of server-side logs is now working.  Try out the new log viewer!  Also a cool note:  The logging plugin implements a pretty novel method for performing asynchronous non-blocking off-loading of CPU-intensive tasks using the multiprocessing module *with* callbacks that can operate on non-picklable variables and instancemethods!  I have been trying to figure out how to do that for a very long time now and I may have stumbled across a completely new programming pattern (for Python anyway).
CSS Themes:  Made some adjustments so that the input elements in the settings panel don't get cut off (slightly) on the right.
  • Loading branch information
@liftoff
liftoff committed Jan 4, 2012
1 parent 3392481 commit a601e36
Show file tree
Hide file tree
Showing 19 changed files with 1,693 additions and 562 deletions.
80 changes: 36 additions & 44 deletions gateone/gateone.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -641,11 +641,13 @@ class StyleHandler(BaseHandler):
# compelling attack vector.
def get(self):
enum = self.get_argument("enumerate", None)
templates_path = os.path.join(GATEONE_DIR, 'templates')
themes_path = os.path.join(templates_path, 'themes')
colors_path = os.path.join(templates_path, 'term_colors')
if enum:
themes = os.listdir(os.path.join(GATEONE_DIR, 'templates/themes'))
themes = os.listdir(themes_path)
themes = [a.replace('.css', '') for a in themes]
colors = os.listdir(
os.path.join(GATEONE_DIR, 'templates/term_colors'))
colors = os.listdir(colors_path)
colors = [a.replace('.css', '') for a in colors]
self.set_header ('Content-Type', 'application/json')
message = {'themes': themes, 'colors': colors}
Expand All @@ -668,27 +670,27 @@ def get(self):
self.set_header ('Content-Type', 'text/css')
if theme:
try:
theme_path = os.path.join(themes_path, "%s.css" % theme)
self.render(
"templates/themes/%s.css" % theme,
theme_path,
container=container,
prefix=prefix,
colors_256=colors_256
)
except IOError:
# Given theme was not found
logging.error(
_("templates/themes/%s.css was not found" % theme))
logging.error(_("%s was not found" % theme_path))
elif colors:
try:
color_path = os.path.join(colors_path, "%s.css" % colors)
self.render(
"templates/term_colors/%s.css" % colors,
color_path,
container=container,
prefix=prefix
)
except IOError:
# Given theme was not found
logging.error(_(
"templates/term_colors/%s.css was not found" % colors))
logging.error(_("%s was not found" % color_path))

class PluginCSSTemplateHandler(BaseHandler):
"""
Expand All @@ -708,17 +710,19 @@ def get(self):
prefix = self.get_argument("prefix")
plugin = self.get_argument("plugin")
template = self.get_argument("template")
templates_path = os.path.join(GATEONE_DIR, 'templates')
plugin_templates_path = os.path.join(templates_path, plugin)
plugin_template = os.path.join(plugin_templates_path, "%s.css" % plugin)
self.set_header ('Content-Type', 'text/css')
try:
self.render(
"templates/%s/%s" % (plugin, template),
plugin_template,
container=container,
prefix=prefix
)
except IOError:
# The provided plugin/template combination was not found
logging.error(
_("templates/%s/%s was not found" % (plugin, template)))
logging.error(_("%s.css was not found" % plugin_template))

class JSPluginsHandler(BaseHandler):
"""
Expand Down Expand Up @@ -780,6 +784,7 @@ def get_current_user(self):

def open(self):
"""Called when a new WebSocket is opened."""
# TODO: Make it so that idle WebSockets that haven't passed authentication tests get auto-closed within N seconds in order to prevent a DoS scenario where the attacker keeps all possible ports open indefinitely.
# client_id is unique to the browser/client whereas session_id is unique
# to the user. It isn't used much right now but it will be useful in
# the future once more stuff is running over WebSockets.
Expand Down Expand Up @@ -834,12 +839,13 @@ def on_close(self):
logging.debug("on_close()")
user = self.get_current_user()
# Remove all attached callbacks so we're not wasting memory/CPU
for term in SESSIONS[self.session]:
if isinstance(term, int):
multiplex = SESSIONS[self.session][term]['multiplex']
multiplex.remove_all_callbacks(self.callback_id)
term_emulator = multiplex.term
term_emulator.remove_all_callbacks(self.callback_id)
if user['session'] in SESSIONS:
for term in SESSIONS[user['session']]:
if isinstance(term, int):
multiplex = SESSIONS[user['session']][term]['multiplex']
multiplex.remove_all_callbacks(self.callback_id)
term_emulator = multiplex.term
term_emulator.remove_all_callbacks(self.callback_id)
if user and 'upn' in user:
logging.info(
_("WebSocket closed (%s).") % user['upn'])
Expand All @@ -856,9 +862,11 @@ def pong(self, timestamp):

def authenticate(self, settings):
"""
Authenticates the client using the given session (which should be
settings['session']) and returns a list of all running terminals (if
any). If no session is given (null) a new one will be created.
Authenticates the client by first trying to use the 'gateone_user'
cookie or if Gate One is configured to use API authentication it will
use *settings['auth']*. Additionally, it will accept
*settings['container']* and *settings['prefix']* to apply those to the
equivalent properties (self.container and self.prefix).
"""
logging.debug("authenticate(): %s" % settings)
# Make sure the client is authenticated if authentication is enabled
Expand Down Expand Up @@ -969,6 +977,12 @@ def authenticate(self, settings):
message = {'notice': _('AUTHENTICATION ERROR: %s' % e)}
self.write_message(json_encode(message))
return
# Apply the container/prefix settings (if present)
# NOTE: Currently these are only used by the logging plugin
if 'container' in settings:
self.container = settings['container']
if 'prefix' in settings:
self.prefix = settings['prefix']
# This check is to make sure there's no existing session so we don't
# accidentally clobber it.
if self.session not in SESSIONS:
Expand Down Expand Up @@ -1400,27 +1414,6 @@ def debug_terminal(self, term):
print("%s:%s" % (i, "".join(line)))
print(renditions[i])

class OpenLogHandler(BaseHandler):
"""
Handles uploads of user logs and returns them to the client as a basic HTML
page. Essentially, this works around the limitation of an HTML page being
unable to save itself =).
"""
def post(self):
log = self.get_argument("log")
container = self.get_argument("container")
prefix = self.get_argument("prefix")
theme = self.get_argument("theme")
css_file = open('templates/css_%s.css' % theme).read()
css = tornado.template.Template(css_file)
self.render(
"templates/user_log.html",
log=log,
container=container,
prefix=prefix,
css=css.generate(container=container, prefix=prefix)
)

# Thread classes
class TidyThread(threading.Thread):
"""
Expand Down Expand Up @@ -1554,7 +1547,6 @@ def __init__(self, settings):
(r"/ws", TerminalWebSocket),
(r"/auth", AuthHandler),
(r"/style", StyleHandler),
(r"/openlog", OpenLogHandler),
(r"/cssrender", PluginCSSTemplateHandler),
(r"/combined_js", JSPluginsHandler),
(r"/docs/(.*)", tornado.web.StaticFileHandler, {
Expand Down Expand Up @@ -1816,7 +1808,7 @@ def main():
config_defaults.update({'log_file_max_size': 100 * 1024 * 1024}) # 100MB
config_defaults.update({'log_file_num_backups': 10})
config_defaults.update({'log_to_stderr': False})
web_log_path = os.path.join(GATEONE_DIR, logs)
web_log_path = os.path.join(GATEONE_DIR, 'logs')
if not os.path.exists(web_log_path):
mkdir_p(web_log_path)
config_defaults.update(
Expand Down
114 changes: 95 additions & 19 deletions gateone/logviewer.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,17 @@
__author__ = 'Dan McDougall <daniel.mcdougall@liftoffsoftware.com>'

# Import stdlib stuff
import sys, re, gzip
import os, sys, re, gzip
from time import sleep
from datetime import datetime
from optparse import OptionParser

# Import our own stuff
from utils import raw
from gateone import PLUGINS

# 3rd party imports
from tornado.escape import json_encode, json_decode

__doc__ = """\
.. _log_viewer:
Expand Down Expand Up @@ -95,6 +99,9 @@

# Globals
SEPARATOR = u"\U000f0f0f" # The character used to separate frames in the log
RE_OPT_SEQ = re.compile(r'\x1b\]_\;(.+?)(\x07|\x1b\\)', re.MULTILINE)
RE_TITLE_SEQ = re.compile(
r'.*\x1b\][0-2]\;(.+?)(\x07|\x1b\\)', re.DOTALL|re.MULTILINE)

# TODO: Support Fast forward/rewind/pause like Gate One itself.

Expand All @@ -109,7 +116,7 @@ def playback_log(log_path, file_like, show_esc=False):
"""
log = gzip.open(log_path).read().decode('utf-8')
prev_frame_time = None
for i, frame in enumerate(log.split(SEPARATOR)):
for i, frame in enumerate(log.split(SEPARATOR)[1:]): # Skip first frame
try:
frame_time = float(frame[:13]) # First 13 chars is the timestamp
frame = frame[14:] # Skips the colon
Expand Down Expand Up @@ -147,12 +154,6 @@ def escape_escape_seq(text, preserve_renditions=True, rstrip=True):
r'\x1b(.*\x1b\\|[ABCDEFGHIJKLMNOQRSTUVWXYZa-z0-9=]|[()# %*+].)')
csi_sequence = re.compile(r'\x1B\[([?A-Za-z0-9;@:\!]*)([A-Za-z@_])')
esc_rstrip = re.compile('[ \t]+\x1b.+$')
#replacement_map = {
#0: u'␀',
#7: u'␇',
#9: u'␉',
#24: u'␘',
#}
out = u""
esc_buffer = u""
# If this seems confusing it is because text parsing is a black art! ARRR!
Expand Down Expand Up @@ -188,13 +189,88 @@ def escape_escape_seq(text, preserve_renditions=True, rstrip=True):
continue
if rstrip:
# Remove trailing whitespace + trailing ESC sequences
return esc_rstrip.sub('', out)
return esc_rstrip.sub('', out).rstrip()
else: # All these trailers better make for a good movie
return out

def retrieve_first_frame(golog_path):
"""
Retrieves the first frame from the given *golog_path*.
"""
found_first_frame = None
frame = ""
f = gzip.open(golog_path)
while not found_first_frame:
frame += f.read(1)
if frame.decode('UTF-8', "ignore").endswith(SEPARATOR):
# That's it; wrap this up
found_first_frame = True
f.close()
return frame.decode('UTF-8', "ignore").rstrip(SEPARATOR)

def get_metadata(golog_path, user):
"""
Retrieves or creates/adds the metadata inside of the golog at *golog_path*.
NOTE: All logs will need "fixing" the first time they're enumerated since
they won't have an end_date. Fortunately we only need to do this once per
golog.
"""
first_frame = retrieve_first_frame(golog_path)
metadata = {}
if first_frame[14:].startswith('{'):
# This is JSON
metadata = json_decode(first_frame[14:])
if 'end_date' in metadata: # end_date gets added by this func
return metadata # All done
# Sadly, we have to read the whole thing into memory to do this
golog_frames = gzip.open(golog_path).read().decode('UTF-8').split(SEPARATOR)
golog_frames.pop() # Get rid of the last (empty) item
# Getting the start and end dates are easy
start_date = golog_frames[0][:13]
end_date = golog_frames[-2][:13] # The very last is empty
num_frames = len(golog_frames)
version = "1.0"
connect_string = None
if 'ssh' in PLUGINS['py']:
# Try to find the host that was connected to by looking for the SSH
# plugin's special optional escape sequence. It looks like this:
# "\x1b]_;ssh|%s@%s:%s\007"
for frame in golog_frames[:50]: # Only look in the first 50 frames
match_obj = RE_OPT_SEQ.match(frame)
if match_obj:
connect_string = match_obj.group(1).split('|')[1]
break
if not connect_string:
# Try guessing it by looking for a title escape sequence
for frame in golog_frames[:50]: # Only look in the first 50 frames
match_obj = RE_TITLE_SEQ.match(frame)
if match_obj:
# The split() here is an attempt to remove titles like this:
# 'someuser@somehost: ~'
connect_string = match_obj.group(1).split(':')[0]
break
metadata.update({
'user': user,
'start_date': start_date,
'end_date': end_date,
'frames': num_frames,
'version': version,
'connect_string': connect_string,
'filename': os.path.split(golog_path)[1]
})
first_frame = "%s:%s%s" % (start_date, json_encode(metadata), SEPARATOR)
# Insert the new first frame
golog_frames.insert(0, first_frame)
# Save it
f = gzip.open(golog_path, 'w')
f.write(SEPARATOR.join(golog_frames).encode('UTF-8'))
f.close()
return metadata

def flatten_log(log_path, preserve_renditions=True, show_esc=False):
"""
Given a log file at *log_path*, return a list of log lines contained within.
Given a log file at *log_path*, return a str of log lines contained within.
If *preserve_renditions* is True, CSI escape sequences for renditions will
be preserved as-is (e.g. font color, background, etc). This is to make the
Expand All @@ -211,7 +287,7 @@ def flatten_log(log_path, preserve_renditions=True, show_esc=False):
import gzip
lines = gzip.open(log_path).read().decode('utf-8')
out = ""
for frame in lines.split(SEPARATOR):
for frame in lines.split(SEPARATOR)[1:]: # Skip the first frame (metadata)
try:
frame_time = float(frame[:13]) # First 13 chars is the timestamp
# Convert to datetime object
Expand All @@ -224,28 +300,28 @@ def flatten_log(log_path, preserve_renditions=True, show_esc=False):
# start of each line in case the previous line didn't
# reset it on its own.
if show_esc:
out += "%s %s\n" % ( # Standard Unix log format
out += u"%s %s\n" % ( # Standard Unix log format
frame_time.strftime(u'\x1b[m%b %m %H:%M:%S'),
raw(fl))
else:
out += "%s %s\n" % ( # Standard Unix log format
out += u"%s %s\n" % ( # Standard Unix log format
frame_time.strftime(u'\x1b[m%b %m %H:%M:%S'),
escape_escape_seq(fl, rstrip=True)
escape_escape_seq(fl, rstrip=True).rstrip()
)
elif i:# Don't need this for the first empty line in a frame
out += frame_time.strftime(u'\x1b[m%b %m %H:%M:%S \n')
elif show_esc:
if len(out) and out[-1] == '\n':
if len(out) and out[-1] == u'\n':
out = u"%s%s\n" % (out[:-1], raw(frame[14:]))
else:
escaped_frame = escape_escape_seq(frame[14:], rstrip=False)
if len(out) and out[-1] == '\n':
escaped_frame = escape_escape_seq(frame[14:], rstrip=True).rstrip()
if len(out) and out[-1] == u'\n':
out = u"%s%s\n" % (out[:-1], escaped_frame)
elif escaped_frame:
# This is pretty much always going to be the first line
out += "%s %s\n" % ( # Standard Unix log format
out += u"%s %s\n" % ( # Standard Unix log format
frame_time.strftime(u'\x1b[m%b %m %H:%M:%S'),
escaped_frame
escaped_frame.rstrip()
)
except ValueError as e:
pass
Expand Down
Loading

0 comments on commit a601e36

Please sign in to comment.