Permalink
Browse files

KLL Compiler Re-Write

This was many months of efforts in re-designing how the KLL compiler should work.
The major problem with the original compiler was how difficult it was to extend language wise.
This lead to many delays in KLL 0.4 and 0.5 being implemented.

The new design is a multi-staged compiler, where even tokenization occurs over multiple stages.
This allows individual parsing and token regexes to be expressed more simply without affect other expressions.

Another area of change is the concept of Contexts.
In the original KLL compiler the idea of a cache assigned was "hacked" on when I realized the language was "broken" (after nearly finishing the compiler).
Since assignment order is generally considered not to matter for keymappings, I created a "cached" assignment where the whole file is read into a sub-datastructure, then apply to the master datastructure.
Unfortunately, this wasn't really all that clear, so it was annoying to work with.
To remedy this, I created KLL Contexts, which contain information about a group of expressions.
Not only can these groups can be merged with other Contexts, they have historical data about how they were generated allowing for errors very late in processing to be pin-pointed back to the offending kll file.

Backends work nearly the same as they did before.
However, all call-backs for capability evaluations have been removed.
This makes the interface much cleaner as Contexts can only be symbolically merged now.
(Previously datastructures did evaluation merges where the ScanCode or Capability was looked up right before passing to the backend, but this required additional information from the backend).

Many of the old parsing and tokenization rules have been reused, along with the hid_dict.py code.

The new design takes advantage of processor pools to handle multithreading where it makes sense.
For example, all specified files are loaded into ram simulatenously rather than sparingly reading from.
The reason for this is so that each Context always has all the information it requires at all times.

kll
- Program entry point (previously kll.py)
- Very small now, does some setting up of command-line args
- Most command-line args are specified by the corresponding processing stage

common/channel.py
- Pixel Channel container classes

common/context.py
- Context container classes
- As is usual with other files, blank classes inherit a base class
- These blank classes are identified by the class name itself to handle special behaviour
- And if/when necessary functions are re-implemented
- MergeConext class facilitates merging of contexts while maintaining lineage

common/expression.py
- Expression container classes
  * Expression base class
  * AssignmentExpression
  * NameAssociationExpression
  * DataAssociationExpression
  * MapExpression
- These classes are used to store expressions after they have finished parsing and tokenization

common/file.py
- Container class for files being read by the KLL compiler

common/emitter.py
- Base class for all KLL emitters
- TextEmitter for dealing with text file templates

common/hid_dict.py
- Slightly modified version of kll_lib/hid_dict.py

common/id.py
- Identification container classes
- Used to indentify different types of elements used within the KLL language

common/modifier.py
- Container classes for animation and pixel change functions

common/organization.py
- Data structure merging container classes
- Contains all the sub-datastructure classes as well
- The Organization class handles the merge orchestration and expression insertion

common/parse.py
- Parsing rules for funcparserlib
- Much of this file was taken from the original kll.py
- Many changes to support the multi-stage processing and support KLL 0.5

common/position.py
- Container class dealing with physical positions

common/schedule.py
- Container class dealing with scheduling and timing events

common/stage.py
- Contains ControlStage and main Stage classes
  * CompilerConfigurationStage
  * FileImportStage
  * PreprocessorStage
  * OperationClassificationStage
  * OperationSpecificsStage
  * OperationOrganizationStage
  * DataOrganziationStage
  * DataFinalizationStage
  * DataAnalysisStage
  * CodeGenerationStage
  * ReportGenerationStage
- Each of these classes controls the life-cycle of each stage
- If multi-threading is desired, it must be handled within the class
  * The next stage will not start until the current stage is finished
- Errors are handled such that as many errors as possible are recorded before forcing an exit
  * The exit is handled at the end of each stage if necessary
- Command-line arguments for each stage can be defined if necessary (they are given their own grouping)
- Each stage can pull variables and functions from other stages if necessary using a name lookup
  * This means you don't have to worry about over-arching datastructures

emitters/emitters.py
- Container class for KLL emitters
- Handles emitter setup and selection

emitters/kiibohd/kiibohd.py
- kiibohd .h file KLL emitter
- Re-uses some backend code from the original KLL compiler

funcparserlib/parser.py
- Added debug mode control

examples/assignment.kll
examples/defaultMapExample.kll
examples/example.kll
examples/hhkbpro2.kll
examples/leds.kll
examples/mapping.kll
examples/simple1.kll
examples/simple2.kll
examples/simpleExample.kll
examples/state_scheduling.kll
- Updating/Adding rules for new compiler and KLL 0.4 + KLL 0.5 support
  • Loading branch information...
haata committed Sep 2, 2016
1 parent c2a798f commit f610d0fb150de0d5ac795a84e8dd122b419e45cd
No changes.
@@ -0,0 +1,75 @@
#!/usr/bin/env python3
'''
KLL Channel Containers
'''
# Copyright (C) 2016 by Jacob Alexander
#
# This file is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This file is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this file. If not, see <http://www.gnu.org/licenses/>.
### Imports ###
### Decorators ###
## Print Decorator Variables
ERROR = '\033[5;1;31mERROR\033[0m:'
WARNING = '\033[5;1;33mWARNING\033[0m:'
### Classes ###
class Channel:
'''
Pixel Channel Container
'''
def __init__( self, uid, width ):
self.uid = uid
self.width = width
def __repr__( self ):
return "{0}:{1}".format( self.uid, self.width )
class ChannelList:
'''
Pixel Channel List Container
'''
def __init__( self ):
self.channels = []
def setChannels( self, channel_list ):
'''
Apply channels to Pixel
'''
for channel in channel_list:
self.channels.append( Channel( channel[0], channel[1] ) )
def strChannels( self ):
'''
__repr__ of Channel when multiple inheritance is used
'''
output = ""
for index, channel in enumerate( self.channels ):
if index > 0:
output += ","
output += "{0}".format( channel )
return output
def __repr__( self ):
return self.strChannels()
@@ -0,0 +1,256 @@
#!/usr/bin/env python3
'''
KLL Context Definitions
* Generic (auto-detected)
* Configuration
* BaseMap
* DefaultMap
* PartialMap
'''
# Copyright (C) 2016 by Jacob Alexander
#
# This file is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This file is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this file. If not, see <http://www.gnu.org/licenses/>.
### Imports ###
import copy
import os
import common.organization as organization
### Decorators ###
## Print Decorator Variables
ERROR = '\033[5;1;31mERROR\033[0m:'
WARNING = '\033[5;1;33mWARNING\033[0m:'
### Classes ###
class Context:
'''
Base KLL Context Class
'''
def __init__( self ):
'''
Context initialization
'''
# Each context may have one or more included kll files
# And each of these files will have at least 1 Context
self.kll_files = []
# File data assigned to each context
# This info is populated during the PreprocessorStage
self.lines = []
self.data = ""
self.parent = None
# Tokenized data sets
self.classification_token_data = []
self.expressions = []
# Organized Expression Datastructure
self.organization = organization.Organization()
def __repr__( self ):
# Build list of all the info
return "(kll_files={0}, lines={1}, data='''{2}''')".format(
self.kll_files,
self.lines,
self.data,
)
def initial_context( self, lines, data, parent ):
'''
Used in the PreprocessorStage to update the initial line and kll file data
@param lines: Data split per line
@param data: Entire context in a single string
@param parent: Parent node, always a KLLFile
'''
self.lines = lines
self.data = data
self.parent = parent
def query( self, kll_expression, kll_type ):
'''
Query
Returns a dictionary of the specified property.
Most queries should use this.
See organization.py:Organization for property_type details.
@param kll_expression: String name of expression type
@param kll_type: String name of the expression sub-type
@return: context_name: (dictionary)
'''
return self.organization.data_mapping[ kll_expression ][ kll_type ]
class GenericContext( Context ):
'''
Generic KLL Context Class
'''
class ConfigurationContext( Context ):
'''
Configuration KLL Context Class
'''
class BaseMapContext( Context ):
'''
Base Map KLL Context Class
'''
class DefaultMapContext( Context ):
'''
Default Map KLL Context Class
'''
class PartialMapContext( Context ):
'''
Partial Map KLL Context Class
'''
def __init__( self, layer ):
'''
Partial Map Layer Context Initialization
@param: Layer associated with Partial Map
'''
super().__init__()
self.layer = layer
class MergeContext( Context ):
'''
Container class for a merged Context
Has references to the original contexts merged in
'''
def __init__( self, base_context ):
'''
Initialize the MergeContext with the base context
Another MergeContext can be used as the base_context
@param base_context: Context used to seed the MergeContext
'''
super().__init__()
# List of context, in the order of merging, starting from the base
self.contexts = [ base_context ]
# Copy the base context Organization into the MergeContext
self.organization = copy.copy( base_context.organization )
# Set the layer if the base is a PartialMapContext
if base_context.__class__.__name__ == 'PartialMapContext':
self.layer = base_context.layer
def merge( self, merge_in, debug ):
'''
Merge in context
Another MergeContext can be merged into a MergeContext
@param merge_in: Context to merge in to this one
@param debug: Enable debug out
'''
# Append to context list
self.contexts.append( merge_in )
# Merge context
self.organization.merge(
merge_in.organization,
debug
)
# Set the layer if the base is a PartialMapContext
if merge_in.__class__.__name__ == 'PartialMapContext':
self.layer = merge_in.layer
def reduction( self ):
'''
Simplifies datastructure
NOTE: This will remove data, therefore, context is lost
'''
self.organization.reduction()
def paths( self ):
'''
Returns list of file paths used to generate this context
'''
file_paths = []
for kll_context in self.contexts:
# If context is a MergeContext then we have to recursively search
if kll_context.__class__.__name__ is 'MergeContext':
file_paths.extend( kll_context.paths() )
else:
file_paths.append( kll_context.parent.path )
return file_paths
def files( self ):
'''
Short form list of file paths used to generate this context
'''
file_paths = []
for file_path in self.paths():
file_paths.append( os.path.basename( file_path ) )
return file_paths
def __repr__( self ):
return "(kll_files={0}, organization={1})".format(
self.files(),
self.organization,
)
def query_contexts( self, kll_expression, kll_type ):
'''
Context Query
Returns a list of tuples (dictionary, kll_context) doing a deep search to the context leaf nodes.
This results in pre-merge data and is useful for querying information about files used during compilation.
See organization.py:Organization for property_type details.
@param kll_expression: String name of expression type
@param kll_type: String name of the expression sub-type
@return: context_name: (dictionary, kll_context)
'''
# Build list of leaf contexts
leaf_contexts = []
for kll_context in self.contexts:
# Recursively search if necessary
if kll_context.__class__.__name__ == 'MergeContext':
leaf_contexts.extend( kll_context.query_contexts( kll_expression, kll_type ) )
else:
leaf_contexts.append( ( kll_context.query( kll_expression, kll_type ), kll_context ) )
return leaf_contexts
Oops, something went wrong.

0 comments on commit f610d0f

Please sign in to comment.