Permalink
Browse files

gateone.py: Removed OpenLogHandler since it isn't being used anymore.

gateone.py:  Fixed the typo in the new default web server logs location.  This should resolve #81
gateone.py:  A few more changes here and there to use os.path.join() instead of assuming the separator is "/".
gateone.py:  The container and prefix are now stored as instance variables of TerminalWebSocket (so they could be used by the Logging plugin).
termio.py:  The log format now includes the number of rows/cols in the metadata.
termio.py:  Made some minor adjustments to the blocked_io_handler (and related checks/functions) to make that whole watchdog-style mechanism more efficient.  Seems to kick in a lot faster now when someone runs a command like "yes".
gateone.js:  Minor change to loadWebWorkerAction() to prevent an exception that can occur when closing terminals (related to the now-gone terminal's title).
gateone.js:  Added a new function to GateOne.Utils:  getOffset().  Does what you'd think.
gateone.js:  The container and prefix are now sent along with the authentication JSON message.
gateone.js:  GateOne.Terminal has been placed inside its own sandbox in order to avoid a circular reference with the GateOne object when executing terminal refreshes.  NOTE: According to the debugger it seems to have worked (no more circular GateOne object references).  Also according to the debugger it seems to have made absolutely no difference in memory consumption.
gateone.js:  Changed how terminal updates are processed slightly in order to avoid creating circular references that could prevent proper JavaScript garbage collection.  In theory, these changes are for the better but I didn't notice much of a change in terms of memory utilization.  Also, Chrome 16 has a memory leak of some sort (with Gate One) that I couldn't seem to work around (which is what I was trying to do with these changes).  Fortunately this problem appears to be fixed in Chrome 17 and you can always reclaim memory by simply reloading Gate One in your browser window.
gateone.js:  Added the ability to disable the bell sound (a visual indication will still appear).  This should resolve #80
logviewer.py:  Moved some of the functions from logging_plugin.py into logviewer.py since it makes more sense to have them there.
terminal.py:  Minor adjustment to the title sequence and special optional escape sequence regular expressions so they won't get confused and have you stuck in a mode where the terminal emulator thinks you're endlessly entering a title or a special optional escape sequence.
utils.py:  ^J and ^M have been commented out of REPLACEMENT_DICT because they were causing issues when playing back logs (which sort of defeated the purpose of that dict in the first place).
Bookmarks Plugin:  Minor change to bookmarks.js to prevent a possible (transient) exception when the iconQueue isn't present while loading.  Also cleaned up a few variables that weren't being used.
Playback Plugin:  Playback of session recordings is now working (again).  I also fixed the logic that will re-size the terminal to fit within the current window.
Logging Plugin:  BIG CHANGE/NEW:  Viewing and playback of server-side logs is now working.  Try out the new log viewer!  Also a cool note:  The logging plugin implements a pretty novel method for performing asynchronous non-blocking off-loading of CPU-intensive tasks using the multiprocessing module *with* callbacks that can operate on non-picklable variables and instancemethods!  I have been trying to figure out how to do that for a very long time now and I may have stumbled across a completely new programming pattern (for Python anyway).
CSS Themes:  Made some adjustments so that the input elements in the settings panel don't get cut off (slightly) on the right.
  • Loading branch information...
1 parent 3392481 commit a601e36f1c736f39afd518e5f8f7c70bdd581231 @liftoff committed Jan 4, 2012
View
@@ -641,11 +641,13 @@ class StyleHandler(BaseHandler):
# compelling attack vector.
def get(self):
enum = self.get_argument("enumerate", None)
+ templates_path = os.path.join(GATEONE_DIR, 'templates')
+ themes_path = os.path.join(templates_path, 'themes')
+ colors_path = os.path.join(templates_path, 'term_colors')
if enum:
- themes = os.listdir(os.path.join(GATEONE_DIR, 'templates/themes'))
+ themes = os.listdir(themes_path)
themes = [a.replace('.css', '') for a in themes]
- colors = os.listdir(
- os.path.join(GATEONE_DIR, 'templates/term_colors'))
+ colors = os.listdir(colors_path)
colors = [a.replace('.css', '') for a in colors]
self.set_header ('Content-Type', 'application/json')
message = {'themes': themes, 'colors': colors}
@@ -668,27 +670,27 @@ def get(self):
self.set_header ('Content-Type', 'text/css')
if theme:
try:
+ theme_path = os.path.join(themes_path, "%s.css" % theme)
self.render(
- "templates/themes/%s.css" % theme,
+ theme_path,
container=container,
prefix=prefix,
colors_256=colors_256
)
except IOError:
# Given theme was not found
- logging.error(
- _("templates/themes/%s.css was not found" % theme))
+ logging.error(_("%s was not found" % theme_path))
elif colors:
try:
+ color_path = os.path.join(colors_path, "%s.css" % colors)
self.render(
- "templates/term_colors/%s.css" % colors,
+ color_path,
container=container,
prefix=prefix
)
except IOError:
# Given theme was not found
- logging.error(_(
- "templates/term_colors/%s.css was not found" % colors))
+ logging.error(_("%s was not found" % color_path))
class PluginCSSTemplateHandler(BaseHandler):
"""
@@ -708,17 +710,19 @@ def get(self):
prefix = self.get_argument("prefix")
plugin = self.get_argument("plugin")
template = self.get_argument("template")
+ templates_path = os.path.join(GATEONE_DIR, 'templates')
+ plugin_templates_path = os.path.join(templates_path, plugin)
+ plugin_template = os.path.join(plugin_templates_path, "%s.css" % plugin)
self.set_header ('Content-Type', 'text/css')
try:
self.render(
- "templates/%s/%s" % (plugin, template),
+ plugin_template,
container=container,
prefix=prefix
)
except IOError:
# The provided plugin/template combination was not found
- logging.error(
- _("templates/%s/%s was not found" % (plugin, template)))
+ logging.error(_("%s.css was not found" % plugin_template))
class JSPluginsHandler(BaseHandler):
"""
@@ -780,6 +784,7 @@ def get_current_user(self):
def open(self):
"""Called when a new WebSocket is opened."""
+ # TODO: Make it so that idle WebSockets that haven't passed authentication tests get auto-closed within N seconds in order to prevent a DoS scenario where the attacker keeps all possible ports open indefinitely.
# client_id is unique to the browser/client whereas session_id is unique
# to the user. It isn't used much right now but it will be useful in
# the future once more stuff is running over WebSockets.
@@ -834,12 +839,13 @@ def on_close(self):
logging.debug("on_close()")
user = self.get_current_user()
# Remove all attached callbacks so we're not wasting memory/CPU
- for term in SESSIONS[self.session]:
- if isinstance(term, int):
- multiplex = SESSIONS[self.session][term]['multiplex']
- multiplex.remove_all_callbacks(self.callback_id)
- term_emulator = multiplex.term
- term_emulator.remove_all_callbacks(self.callback_id)
+ if user['session'] in SESSIONS:
+ for term in SESSIONS[user['session']]:
+ if isinstance(term, int):
+ multiplex = SESSIONS[user['session']][term]['multiplex']
+ multiplex.remove_all_callbacks(self.callback_id)
+ term_emulator = multiplex.term
+ term_emulator.remove_all_callbacks(self.callback_id)
if user and 'upn' in user:
logging.info(
_("WebSocket closed (%s).") % user['upn'])
@@ -856,9 +862,11 @@ def pong(self, timestamp):
def authenticate(self, settings):
"""
- Authenticates the client using the given session (which should be
- settings['session']) and returns a list of all running terminals (if
- any). If no session is given (null) a new one will be created.
+ Authenticates the client by first trying to use the 'gateone_user'
+ cookie or if Gate One is configured to use API authentication it will
+ use *settings['auth']*. Additionally, it will accept
+ *settings['container']* and *settings['prefix']* to apply those to the
+ equivalent properties (self.container and self.prefix).
"""
logging.debug("authenticate(): %s" % settings)
# Make sure the client is authenticated if authentication is enabled
@@ -969,6 +977,12 @@ def authenticate(self, settings):
message = {'notice': _('AUTHENTICATION ERROR: %s' % e)}
self.write_message(json_encode(message))
return
+ # Apply the container/prefix settings (if present)
+ # NOTE: Currently these are only used by the logging plugin
+ if 'container' in settings:
+ self.container = settings['container']
+ if 'prefix' in settings:
+ self.prefix = settings['prefix']
# This check is to make sure there's no existing session so we don't
# accidentally clobber it.
if self.session not in SESSIONS:
@@ -1400,27 +1414,6 @@ def debug_terminal(self, term):
print("%s:%s" % (i, "".join(line)))
print(renditions[i])
-class OpenLogHandler(BaseHandler):
- """
- Handles uploads of user logs and returns them to the client as a basic HTML
- page. Essentially, this works around the limitation of an HTML page being
- unable to save itself =).
- """
- def post(self):
- log = self.get_argument("log")
- container = self.get_argument("container")
- prefix = self.get_argument("prefix")
- theme = self.get_argument("theme")
- css_file = open('templates/css_%s.css' % theme).read()
- css = tornado.template.Template(css_file)
- self.render(
- "templates/user_log.html",
- log=log,
- container=container,
- prefix=prefix,
- css=css.generate(container=container, prefix=prefix)
- )
-
# Thread classes
class TidyThread(threading.Thread):
"""
@@ -1554,7 +1547,6 @@ def __init__(self, settings):
(r"/ws", TerminalWebSocket),
(r"/auth", AuthHandler),
(r"/style", StyleHandler),
- (r"/openlog", OpenLogHandler),
(r"/cssrender", PluginCSSTemplateHandler),
(r"/combined_js", JSPluginsHandler),
(r"/docs/(.*)", tornado.web.StaticFileHandler, {
@@ -1816,7 +1808,7 @@ def main():
config_defaults.update({'log_file_max_size': 100 * 1024 * 1024}) # 100MB
config_defaults.update({'log_file_num_backups': 10})
config_defaults.update({'log_to_stderr': False})
- web_log_path = os.path.join(GATEONE_DIR, logs)
+ web_log_path = os.path.join(GATEONE_DIR, 'logs')
if not os.path.exists(web_log_path):
mkdir_p(web_log_path)
config_defaults.update(
View
@@ -11,13 +11,17 @@
__author__ = 'Dan McDougall <daniel.mcdougall@liftoffsoftware.com>'
# Import stdlib stuff
-import sys, re, gzip
+import os, sys, re, gzip
from time import sleep
from datetime import datetime
from optparse import OptionParser
# Import our own stuff
from utils import raw
+from gateone import PLUGINS
+
+# 3rd party imports
+from tornado.escape import json_encode, json_decode
__doc__ = """\
.. _log_viewer:
@@ -95,6 +99,9 @@
# Globals
SEPARATOR = u"\U000f0f0f" # The character used to separate frames in the log
+RE_OPT_SEQ = re.compile(r'\x1b\]_\;(.+?)(\x07|\x1b\\)', re.MULTILINE)
+RE_TITLE_SEQ = re.compile(
+ r'.*\x1b\][0-2]\;(.+?)(\x07|\x1b\\)', re.DOTALL|re.MULTILINE)
# TODO: Support Fast forward/rewind/pause like Gate One itself.
@@ -109,7 +116,7 @@ def playback_log(log_path, file_like, show_esc=False):
"""
log = gzip.open(log_path).read().decode('utf-8')
prev_frame_time = None
- for i, frame in enumerate(log.split(SEPARATOR)):
+ for i, frame in enumerate(log.split(SEPARATOR)[1:]): # Skip first frame
try:
frame_time = float(frame[:13]) # First 13 chars is the timestamp
frame = frame[14:] # Skips the colon
@@ -147,12 +154,6 @@ def escape_escape_seq(text, preserve_renditions=True, rstrip=True):
r'\x1b(.*\x1b\\|[ABCDEFGHIJKLMNOQRSTUVWXYZa-z0-9=]|[()# %*+].)')
csi_sequence = re.compile(r'\x1B\[([?A-Za-z0-9;@:\!]*)([A-Za-z@_])')
esc_rstrip = re.compile('[ \t]+\x1b.+$')
- #replacement_map = {
- #0: u'␀',
- #7: u'␇',
- #9: u'␉',
- #24: u'␘',
- #}
out = u""
esc_buffer = u""
# If this seems confusing it is because text parsing is a black art! ARRR!
@@ -188,13 +189,88 @@ def escape_escape_seq(text, preserve_renditions=True, rstrip=True):
continue
if rstrip:
# Remove trailing whitespace + trailing ESC sequences
- return esc_rstrip.sub('', out)
+ return esc_rstrip.sub('', out).rstrip()
else: # All these trailers better make for a good movie
return out
+def retrieve_first_frame(golog_path):
+ """
+ Retrieves the first frame from the given *golog_path*.
+ """
+ found_first_frame = None
+ frame = ""
+ f = gzip.open(golog_path)
+ while not found_first_frame:
+ frame += f.read(1)
+ if frame.decode('UTF-8', "ignore").endswith(SEPARATOR):
+ # That's it; wrap this up
+ found_first_frame = True
+ f.close()
+ return frame.decode('UTF-8', "ignore").rstrip(SEPARATOR)
+
+def get_metadata(golog_path, user):
+ """
+ Retrieves or creates/adds the metadata inside of the golog at *golog_path*.
+
+ NOTE: All logs will need "fixing" the first time they're enumerated since
+ they won't have an end_date. Fortunately we only need to do this once per
+ golog.
+ """
+ first_frame = retrieve_first_frame(golog_path)
+ metadata = {}
+ if first_frame[14:].startswith('{'):
+ # This is JSON
+ metadata = json_decode(first_frame[14:])
+ if 'end_date' in metadata: # end_date gets added by this func
+ return metadata # All done
+ # Sadly, we have to read the whole thing into memory to do this
+ golog_frames = gzip.open(golog_path).read().decode('UTF-8').split(SEPARATOR)
+ golog_frames.pop() # Get rid of the last (empty) item
+ # Getting the start and end dates are easy
+ start_date = golog_frames[0][:13]
+ end_date = golog_frames[-2][:13] # The very last is empty
+ num_frames = len(golog_frames)
+ version = "1.0"
+ connect_string = None
+ if 'ssh' in PLUGINS['py']:
+ # Try to find the host that was connected to by looking for the SSH
+ # plugin's special optional escape sequence. It looks like this:
+ # "\x1b]_;ssh|%s@%s:%s\007"
+ for frame in golog_frames[:50]: # Only look in the first 50 frames
+ match_obj = RE_OPT_SEQ.match(frame)
+ if match_obj:
+ connect_string = match_obj.group(1).split('|')[1]
+ break
+ if not connect_string:
+ # Try guessing it by looking for a title escape sequence
+ for frame in golog_frames[:50]: # Only look in the first 50 frames
+ match_obj = RE_TITLE_SEQ.match(frame)
+ if match_obj:
+ # The split() here is an attempt to remove titles like this:
+ # 'someuser@somehost: ~'
+ connect_string = match_obj.group(1).split(':')[0]
+ break
+ metadata.update({
+ 'user': user,
+ 'start_date': start_date,
+ 'end_date': end_date,
+ 'frames': num_frames,
+ 'version': version,
+ 'connect_string': connect_string,
+ 'filename': os.path.split(golog_path)[1]
+ })
+ first_frame = "%s:%s%s" % (start_date, json_encode(metadata), SEPARATOR)
+ # Insert the new first frame
+ golog_frames.insert(0, first_frame)
+ # Save it
+ f = gzip.open(golog_path, 'w')
+ f.write(SEPARATOR.join(golog_frames).encode('UTF-8'))
+ f.close()
+ return metadata
+
def flatten_log(log_path, preserve_renditions=True, show_esc=False):
"""
- Given a log file at *log_path*, return a list of log lines contained within.
+ Given a log file at *log_path*, return a str of log lines contained within.
If *preserve_renditions* is True, CSI escape sequences for renditions will
be preserved as-is (e.g. font color, background, etc). This is to make the
@@ -211,7 +287,7 @@ def flatten_log(log_path, preserve_renditions=True, show_esc=False):
import gzip
lines = gzip.open(log_path).read().decode('utf-8')
out = ""
- for frame in lines.split(SEPARATOR):
+ for frame in lines.split(SEPARATOR)[1:]: # Skip the first frame (metadata)
try:
frame_time = float(frame[:13]) # First 13 chars is the timestamp
# Convert to datetime object
@@ -224,28 +300,28 @@ def flatten_log(log_path, preserve_renditions=True, show_esc=False):
# start of each line in case the previous line didn't
# reset it on its own.
if show_esc:
- out += "%s %s\n" % ( # Standard Unix log format
+ out += u"%s %s\n" % ( # Standard Unix log format
frame_time.strftime(u'\x1b[m%b %m %H:%M:%S'),
raw(fl))
else:
- out += "%s %s\n" % ( # Standard Unix log format
+ out += u"%s %s\n" % ( # Standard Unix log format
frame_time.strftime(u'\x1b[m%b %m %H:%M:%S'),
- escape_escape_seq(fl, rstrip=True)
+ escape_escape_seq(fl, rstrip=True).rstrip()
)
elif i:# Don't need this for the first empty line in a frame
out += frame_time.strftime(u'\x1b[m%b %m %H:%M:%S \n')
elif show_esc:
- if len(out) and out[-1] == '\n':
+ if len(out) and out[-1] == u'\n':
out = u"%s%s\n" % (out[:-1], raw(frame[14:]))
else:
- escaped_frame = escape_escape_seq(frame[14:], rstrip=False)
- if len(out) and out[-1] == '\n':
+ escaped_frame = escape_escape_seq(frame[14:], rstrip=True).rstrip()
+ if len(out) and out[-1] == u'\n':
out = u"%s%s\n" % (out[:-1], escaped_frame)
elif escaped_frame:
# This is pretty much always going to be the first line
- out += "%s %s\n" % ( # Standard Unix log format
+ out += u"%s %s\n" % ( # Standard Unix log format
frame_time.strftime(u'\x1b[m%b %m %H:%M:%S'),
- escaped_frame
+ escaped_frame.rstrip()
)
except ValueError as e:
pass
Oops, something went wrong.

0 comments on commit a601e36

Please sign in to comment.