-
Notifications
You must be signed in to change notification settings - Fork 12
/
tokenise_qpopulated.go
65 lines (49 loc) · 2.06 KB
/
tokenise_qpopulated.go
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
// vi: sw=4 ts=4:
/*
---------------------------------------------------------------------------
Copyright (c) 2013-2015 AT&T Intellectual Property
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at:
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
---------------------------------------------------------------------------
*/
package token
import (
)
/*
---------------------------------------------------------------------------------------------
Mnemonic: tokenise_qpopulated
Returns: number of tokens, tokens[]
Date: 22 Apr 2012
Author: E. Scott Daniels
Mods: 13 Jan 2014 - corrected bug causing quotes to be left on last token if
there isn't a sep inside of the tokens.
02 May 2014 - corrected documentation, removed unneeded commented out code.
30 Nov 2014 - Now allows escaped quotes in quoted portion.
01 Dec 2014 - Now uses the work horse tokenise_all() function, so the majority
of this disappears.
---------------------------------------------------------------------------------------------
*/
/*
Takes a string and slices it into tokens using the characters in sepchrs
as the breaking points, but allowing double quotes to provide protection
against separation. For example, if sepchrs is ",|", then the string
foo,bar,"hello,world",,"you|me"
would break into 4 tokens:
foo
bar
hello,world
you|me
Similar to tokenise_qsep, but this method removes empty tokens from the
final result.
The return value is the number of tokens and the list of tokens.
*/
func Tokenise_qpopulated( buf string, sepchrs string ) (int, []string) {
return tokenise_all( buf, sepchrs, false )
}