Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Newer
Older
100644 215 lines (154 sloc) 12.457 kb
84598f7 @tomaz Implemented unit testing target and basic class parsing tests.
authored
1 //
2 // GBTokenizer.h
3 // appledoc
4 //
5 // Created by Tomaz Kragelj on 25.7.10.
6 // Copyright (C) 2010, Gentle Bytes. All rights reserved.
7 //
8
9 #import <Foundation/Foundation.h>
10 #import "ParseKit.h"
11
0dd2364 @tomaz Renamed GBDeclaredFileData to GBSourceInfo to better convey the purpose ...
authored
12 @class GBSourceInfo;
9047fcf @tomaz Refactored comment handling with GBModelBase by removing registerComment...
authored
13 @class GBComment;
6cfd48f @tomaz Refactored file data creation by moving the method to GBTokenizer.
authored
14
84598f7 @tomaz Implemented unit testing target and basic class parsing tests.
authored
15 /** Provides common methods for tokenizing input source strings.
16
0bb1c7d @tomaz Enabled line wrapping and updated comments to fit better.
authored
17 Main responsibilities of the class are to split the given source string into tokens and provide simple methods for iterating over the tokens stream. It works upon ParseKit framework's `PKTokenizer`. As different parsers require different tokenizers and setups, the class itself doesn't create a tokenizer, but instead requires the client to provide one. Here's an example of simple usage:
84598f7 @tomaz Implemented unit testing target and basic class parsing tests.
authored
18
2796c5a @tomaz Added filename information to GBTokenizer to make it accessible for all ...
authored
19 NSString *filename = ...
84598f7 @tomaz Implemented unit testing target and basic class parsing tests.
authored
20 NSString *input = ...
21 PKTokenizer *worker = [PKTokenizer tokenizerWithString:input];
2796c5a @tomaz Added filename information to GBTokenizer to make it accessible for all ...
authored
22 GBTokenizer *tokenizer = [[GBTokenizer allow] initWithTokenizer:worker filename:filename];
84598f7 @tomaz Implemented unit testing target and basic class parsing tests.
authored
23 while (![tokenizer eof]) {
24 NSLog(@"%@", [tokenizer currentToken]);
25 [tokenizer consume:1];
26 }
27
2796c5a @tomaz Added filename information to GBTokenizer to make it accessible for all ...
authored
28 This example simply iterates over all tokens and prints each one to the log. If you want to parse a block of input with known start and/or end token, you can use one of the block consuming methods instead. Note that you still need to provide the name of the file as this is used for creating `GBSourceInfo` objects for parsed objects!
47e2c01 @tomaz Implemented basic comments parsing and tests.
authored
29
4262bb8 @tomaz Fixed comments parsing for non-trivial sources. Closes #43.
authored
30 To make comments parsing simpler, `GBTokenizer` automatically enables comment reporting to the underlying `PKTokenizer`, however to prevent higher level parsers dealing with complexity of comments, any lookahead and consume method doesn't report them. Instead these methods skip all comment tokens, however they do make them accessible through properties, so if the client wants to check whether there's any comment associated with current token, it can simply ask by sending `lastCommentString`. Additionally, the client can also get the value of a comment just before the last one by sending `previousCommentString` - this can be used to get any method section comments which aren't associated with any element. If there is no "stand-alone" comment before the last one, `previousCommentString` returns `nil`. `GBTokenizer` goes even further when dealing with comments - it automatically groups single line comments into a single comment group and removes all prefixes and suffixes.
31
32 @warning *Note:* Both comment values are persistent until a new comment is found! At that time, previous comment contains the value of last comment and the new comment is stored as last comment. This allows us parsing through complex code (like `#ifdef` / `#elif` / `#else` blocks etc.) without fear of loosing any comment information. It does require manual resetting of comments whenever the comment is actually attached to an object. Resetting is performed by sending `resetComments` message to the receiver.
84598f7 @tomaz Implemented unit testing target and basic class parsing tests.
authored
33 */
34 @interface GBTokenizer : NSObject
35
36 ///---------------------------------------------------------------------------------------
37 /// @name Initialization & disposal
38 ///---------------------------------------------------------------------------------------
39
40 /** Returns initialized autoreleased instance using the given source `PKTokenizer`.
41
c68cb86 @tomaz Implemented proper handling of @name sections in more complex circumstan...
authored
42 This should only be used for unit testing. It is equal to sending `tokenizerWithSource:filename:settings:` and passing `nil` for settings. Internally this is exactly how the method is handled.
43
84598f7 @tomaz Implemented unit testing target and basic class parsing tests.
authored
44 @param tokenizer The underlying (worker) tokenizer to use for actual splitting.
a050b89 @tomaz Fixed code documentation thanks to warnings from the appledoc...
authored
45 @param filename The name of the file without path used for generating source info.
84598f7 @tomaz Implemented unit testing target and basic class parsing tests.
authored
46 @return Returns initialized instance or `nil` if failed.
2796c5a @tomaz Added filename information to GBTokenizer to make it accessible for all ...
authored
47 @exception NSException Thrown if the given tokenizer or filename is `nil` or filename is empty string.
84598f7 @tomaz Implemented unit testing target and basic class parsing tests.
authored
48 */
2796c5a @tomaz Added filename information to GBTokenizer to make it accessible for all ...
authored
49 + (id)tokenizerWithSource:(PKTokenizer *)tokenizer filename:(NSString *)filename;
84598f7 @tomaz Implemented unit testing target and basic class parsing tests.
authored
50
c68cb86 @tomaz Implemented proper handling of @name sections in more complex circumstan...
authored
51 /** Returns initialized autoreleased instance using the given source `PKTokenizer`.
52
53 @param tokenizer The underlying (worker) tokenizer to use for actual splitting.
54 @param filename The name of the file without path used for generating source info.
55 @param settings The settings to use for matching comments.
56 @return Returns initialized instance or `nil` if failed.
57 @exception NSException Thrown if the given tokenizer or filename is `nil` or filename is empty string.
58 */
59 + (id)tokenizerWithSource:(PKTokenizer *)tokenizer filename:(NSString *)filename settings:(id)settings;
60
84598f7 @tomaz Implemented unit testing target and basic class parsing tests.
authored
61 /** Initializes tokenizer with the given source `PKTokenizer`.
62
63 This is designated initializer.
64
65 @param tokenizer The underlying (worker) tokenizer to use for actual splitting.
2796c5a @tomaz Added filename information to GBTokenizer to make it accessible for all ...
authored
66 @param filename The name of the file without path that's the source for _tokenizer_'s input string.
c68cb86 @tomaz Implemented proper handling of @name sections in more complex circumstan...
authored
67 @param settings The settings to use for this session.
84598f7 @tomaz Implemented unit testing target and basic class parsing tests.
authored
68 @return Returns initialized instance or `nil` if failed.
2796c5a @tomaz Added filename information to GBTokenizer to make it accessible for all ...
authored
69 @exception NSException Thrown if the given tokenizer or filename is `nil` or filename is empty string.
84598f7 @tomaz Implemented unit testing target and basic class parsing tests.
authored
70 */
c68cb86 @tomaz Implemented proper handling of @name sections in more complex circumstan...
authored
71 - (id)initWithSourceTokenizer:(PKTokenizer *)tokenizer filename:(NSString *)filename settings:(id)settings;
84598f7 @tomaz Implemented unit testing target and basic class parsing tests.
authored
72
73 ///---------------------------------------------------------------------------------------
74 /// @name Tokenizing handling
75 ///---------------------------------------------------------------------------------------
76
77 /** Returns the current token.
47e2c01 @tomaz Implemented basic comments parsing and tests.
authored
78
d6b842b @tomaz Fixed comment warnings for various objects.
authored
79 @see consume:
80 @see lookahead:
84598f7 @tomaz Implemented unit testing target and basic class parsing tests.
authored
81 */
82 - (PKToken *)currentToken;
83
84 /** Returns the token by looking ahead the given number of tokens from current position.
85
0bb1c7d @tomaz Enabled line wrapping and updated comments to fit better.
authored
86 If offset "points" within a valid token, the token is returned, otherwise EOF token is returned. Note that this method automatically skips any comment tokens and only counts actual language tokens.
84598f7 @tomaz Implemented unit testing target and basic class parsing tests.
authored
87
88 @param offset The offset from the current position.
89 @return Returns the token at the given offset or EOF token if offset point after EOF.
d6b842b @tomaz Fixed comment warnings for various objects.
authored
90 @see consume:
3d2d2a8 @groue Added support for termination macros for method declaration and properti...
groue authored
91 @see lookaheadTo:usingBlock:
84598f7 @tomaz Implemented unit testing target and basic class parsing tests.
authored
92 */
93 - (PKToken *)lookahead:(NSUInteger)offset;
94
3d2d2a8 @groue Added support for termination macros for method declaration and properti...
groue authored
95 /** Enumerates but does not consume all tokens starting at current token up until the given end token is detected.
96
97 For each token, the given block is called which gives client a chance to inspect and handle tokens. End token is not reported. Note that this method automatically skips any comment tokens and only enumerates actual language tokens.
98
99 @param end Ending token.
100 @see lookahead:
101 */
102 - (void)lookaheadTo:(NSString *)end usingBlock:(void (^)(PKToken *token, BOOL *stop))block;
103
84598f7 @tomaz Implemented unit testing target and basic class parsing tests.
authored
104 /** Consumes the given ammoun of tokens, starting at the current position.
105
0bb1c7d @tomaz Enabled line wrapping and updated comments to fit better.
authored
106 This effectively "moves" `currentToken` to the new position. If EOF is reached before consuming the given ammount of tokens, consuming stops at the end of stream and `currentToken` returns EOF token. If comment tokens are detected while consuming, they are not counted and consuming count continues with actual language tokens. However if there is a comment just before the next current token (i.e. after the last consumed token), the comment data is saved and is available through `lastCommentString`. Otherwise last comment data is cleared, even if a comment was detected in between.
84598f7 @tomaz Implemented unit testing target and basic class parsing tests.
authored
107
108 @param count The number of tokens to consume.
d6b842b @tomaz Fixed comment warnings for various objects.
authored
109 @see lastComment
110 @see previousComment
84598f7 @tomaz Implemented unit testing target and basic class parsing tests.
authored
111 */
112 - (void)consume:(NSUInteger)count;
113
28a51ce @tomaz Implemented class parsing for class name, superclass name, adopted proto...
authored
114 /** Enumerates and consumes all tokens starting at current token up until the given end token is detected.
115
0bb1c7d @tomaz Enabled line wrapping and updated comments to fit better.
authored
116 For each token, the given block is called which gives client a chance to inspect and handle tokens. End token is not reported and is automatically consumed after all previous tokens are reported. Sending this message is equivalent to sending `consumeFrom:to:usingBlock:` and passing `nil` for start token. Also read `consume:` documentation to understand how comments are dealt with.
28a51ce @tomaz Implemented class parsing for class name, superclass name, adopted proto...
authored
117
118 @param end Ending token.
72ab97c @tomaz Refactored GBTokenizer block consuming method to allow quitting at any t...
authored
119 @param block The block to be called for each token.
28a51ce @tomaz Implemented class parsing for class name, superclass name, adopted proto...
authored
120 @exception NSException Thrown if the given end token is `nil`.
d6b842b @tomaz Fixed comment warnings for various objects.
authored
121 @see lastComment
122 @see previousComment
28a51ce @tomaz Implemented class parsing for class name, superclass name, adopted proto...
authored
123 */
8bebbe8 @tomaz Implemented class declaration parsing and renamed GBTokenizer consume wi...
authored
124 - (void)consumeTo:(NSString *)end usingBlock:(void (^)(PKToken *token, BOOL *consume, BOOL *stop))block;
28a51ce @tomaz Implemented class parsing for class name, superclass name, adopted proto...
authored
125
126 /** Enumerates and consumes all tokens starting at current token up until the given end token is detected.
127
0bb1c7d @tomaz Enabled line wrapping and updated comments to fit better.
authored
128 For each token, the given block is called which gives client a chance to inspect and handle tokens. If start token is given and current token matches it, the token is consumed without reporting it to block. However if the token doesn't match, the method returns immediately without doint anything. End token is also not reported and is also automatically consumed after all previous tokens are reported. Also read `consume:` documentation to understand how comments are dealt with.
28a51ce @tomaz Implemented class parsing for class name, superclass name, adopted proto...
authored
129
130 @param start Optional starting token or `nil`.
131 @param end Ending token.
72ab97c @tomaz Refactored GBTokenizer block consuming method to allow quitting at any t...
authored
132 @param block The block to be called for each token.
28a51ce @tomaz Implemented class parsing for class name, superclass name, adopted proto...
authored
133 @exception NSException Thrown if the given end token is `nil`.
d6b842b @tomaz Fixed comment warnings for various objects.
authored
134 @see lastComment
135 @see previousComment
28a51ce @tomaz Implemented class parsing for class name, superclass name, adopted proto...
authored
136 */
8bebbe8 @tomaz Implemented class declaration parsing and renamed GBTokenizer consume wi...
authored
137 - (void)consumeFrom:(NSString *)start to:(NSString *)end usingBlock:(void (^)(PKToken *token, BOOL *consume, BOOL *stop))block;
28a51ce @tomaz Implemented class parsing for class name, superclass name, adopted proto...
authored
138
84598f7 @tomaz Implemented unit testing target and basic class parsing tests.
authored
139 /** Specifies whether we're at EOF.
140
141 @return Returns `YES` if we're at EOF, `NO` otherwise.
142 */
143 - (BOOL)eof;
144
47e2c01 @tomaz Implemented basic comments parsing and tests.
authored
145 ///---------------------------------------------------------------------------------------
6cfd48f @tomaz Refactored file data creation by moving the method to GBTokenizer.
authored
146 /// @name Information handling
147 ///---------------------------------------------------------------------------------------
148
a050b89 @tomaz Fixed code documentation thanks to warnings from the appledoc...
authored
149 /** Returns `GBSourceInfo` for current token and filename.
6cfd48f @tomaz Refactored file data creation by moving the method to GBTokenizer.
authored
150
d6b842b @tomaz Fixed comment warnings for various objects.
authored
151 This is equivalent to sending `sourceInfoForToken:` and passing `currentToken` as the _token_ parameter.
6cfd48f @tomaz Refactored file data creation by moving the method to GBTokenizer.
authored
152
153 @return Returns declared file data.
2796c5a @tomaz Added filename information to GBTokenizer to make it accessible for all ...
authored
154 @exception NSException Thrown if current token is `nil`.
d6b842b @tomaz Fixed comment warnings for various objects.
authored
155 @see sourceInfoForToken:
6cfd48f @tomaz Refactored file data creation by moving the method to GBTokenizer.
authored
156 */
9a499e4 @tomaz Fixed comment source info handling when two consequtive comments are fou...
authored
157 - (GBSourceInfo *)sourceInfoForCurrentToken;
6cfd48f @tomaz Refactored file data creation by moving the method to GBTokenizer.
authored
158
a050b89 @tomaz Fixed code documentation thanks to warnings from the appledoc...
authored
159 /** Returns `GBSourceInfo` object describing the given token source information.
6cfd48f @tomaz Refactored file data creation by moving the method to GBTokenizer.
authored
160
2796c5a @tomaz Added filename information to GBTokenizer to make it accessible for all ...
authored
161 The method converts the given token's offset within the input string to line number and uses that information together with assigned `filename` to prepare the token info object.
6cfd48f @tomaz Refactored file data creation by moving the method to GBTokenizer.
authored
162
163 @param token The token for which to get file data.
164 @return Returns declared file data.
2796c5a @tomaz Added filename information to GBTokenizer to make it accessible for all ...
authored
165 @exception NSException Thrown if the given token is `nil`.
d6b842b @tomaz Fixed comment warnings for various objects.
authored
166 @see sourceInfoForCurrentToken
6cfd48f @tomaz Refactored file data creation by moving the method to GBTokenizer.
authored
167 */
9a499e4 @tomaz Fixed comment source info handling when two consequtive comments are fou...
authored
168 - (GBSourceInfo *)sourceInfoForToken:(PKToken *)token;
6cfd48f @tomaz Refactored file data creation by moving the method to GBTokenizer.
authored
169
170 ///---------------------------------------------------------------------------------------
47e2c01 @tomaz Implemented basic comments parsing and tests.
authored
171 /// @name Comments handling
172 ///---------------------------------------------------------------------------------------
173
4262bb8 @tomaz Fixed comments parsing for non-trivial sources. Closes #43.
authored
174 /** Resets `lastComment` and `previousComment` values.
175
176 This message should be sent whenever a comment is "attached" to an object. As comments are persistent, failing to reset would lead to using the same comment for next object as well!
177
178 @see lastComment
179 @see previousComment
180 */
181 - (void)resetComments;
182
9047fcf @tomaz Refactored comment handling with GBModelBase by removing registerComment...
authored
183 /** Returns the last comment or `nil` if comment is not available.
47e2c01 @tomaz Implemented basic comments parsing and tests.
authored
184
9047fcf @tomaz Refactored comment handling with GBModelBase by removing registerComment...
authored
185 The returned `[GBComment stringValue]` contains the whole last comment string, without prefixes or suffixes. To optimize things a bit, the actual comment string value is prepared on the fly, as you send the message, so it's only handled if needed. As creating comment string adds some computing overhead, you should cache returned value if possible.
47e2c01 @tomaz Implemented basic comments parsing and tests.
authored
186
187 If there's no comment available for current token, `nil` is returned.
4262bb8 @tomaz Fixed comments parsing for non-trivial sources. Closes #43.
authored
188
189 @see resetComments
9047fcf @tomaz Refactored comment handling with GBModelBase by removing registerComment...
authored
190 @see previousComment
47e2c01 @tomaz Implemented basic comments parsing and tests.
authored
191 */
9047fcf @tomaz Refactored comment handling with GBModelBase by removing registerComment...
authored
192 @property (readonly) GBComment *lastComment;
47e2c01 @tomaz Implemented basic comments parsing and tests.
authored
193
c772576 @tomaz Implemented last stand-alone comment parsing.
authored
194 /** Returns "stand-alone" comment found immediately before the comment returned from `lastCommentString`.
195
9047fcf @tomaz Refactored comment handling with GBModelBase by removing registerComment...
authored
196 Previous comment is a "stand-alone" comment which is found immediately before `lastCommentString` but isn't associated with any language element. These are ussually used to provide meta data and other instructions for formatting or grouping of "normal" comments returned with `lastCommentString`. The value should be used at the same time as `lastCommentString` as it is automatically cleared on the next consuming! If there's no stand-alone comment immediately before last comment, the value returned is `nil`.
c772576 @tomaz Implemented last stand-alone comment parsing.
authored
197
9047fcf @tomaz Refactored comment handling with GBModelBase by removing registerComment...
authored
198 The returned `[GBComment stringValue]` contains the whole previous comment string, without prefixes or suffixes. To optimize things a bit, the actual comment string value is prepared on the fly, as you send the message, so it's only handled if needed. As creating comment string adds some computing overhead, you should cache returned value if possible.
c772576 @tomaz Implemented last stand-alone comment parsing.
authored
199
4262bb8 @tomaz Fixed comments parsing for non-trivial sources. Closes #43.
authored
200 @see resetComments
9047fcf @tomaz Refactored comment handling with GBModelBase by removing registerComment...
authored
201 @see lastComment
c772576 @tomaz Implemented last stand-alone comment parsing.
authored
202 */
9047fcf @tomaz Refactored comment handling with GBModelBase by removing registerComment...
authored
203 @property (readonly) GBComment *previousComment;
c772576 @tomaz Implemented last stand-alone comment parsing.
authored
204
d1c6d80 Fixes issue #38
Daniel Höpfl authored
205
206 /** Returns a "postfix" comment found between the start token and the current token.
207
208 @param startToken start of the current parsing entity.
209 @see lastComment
210 @see previousComment
211 */
212 - (GBComment *)postfixCommentFrom:(PKToken *)startToken;
213
84598f7 @tomaz Implemented unit testing target and basic class parsing tests.
authored
214 @end
Something went wrong with that request. Please try again.