-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Is there a way to use this to decode JSON lines? #33
Comments
there are two reasons why this may be slow: let trimmed = line.trimmingCharacters(in: .whitespaces) this is not necessary. the if you know the root message is an object (and not an array), it may also be slightly faster to use the let log: Log = try .init(from: decoder) this is probably the main reason why your decoding is slow, because it is using the standard library alas i have not gotten a chance to write tutorials for by the way, finally, would you mind sharing your test data (and the toolchain version used to compile the test program)? a 4x slowdown seems extreme, even if you're using the |
Thanks for the quick reply @Kelvin13. I tried removing let (bytes, urlResponse) = try! await URLSession.shared.bytes(from: file)
var logs: [Log] = []
do {
let startTime = CFAbsoluteTimeGetCurrent()
for try await line in bytes.lines {
if !line.isEmpty {
let decoder = try Grammar.parse(line.utf8, as: JSON.Rule<String.Index>.Object.self)
}
}
let timeElapsed = CFAbsoluteTimeGetCurrent() - startTime
print("Time elapsed: \(timeElapsed) s.")
} catch {
print("Reading file failed with error \(error)")
} Side note, how would you create a I am unfortunately not able to share the file as it contains sensitive data, if you need one created I can do so, but for now heres an example file:
I'm not sure if this is what you're looking for:
|
Also 0.3.0 doesn't seem to exist as a tag, but it's in the README so not sure if something has broken there? |
it is very strange that is is taking 85 seconds even with no decoding taking place. can you please regenerate the input data so i can investigate this further? by the way, you are using i don’t know why this is affecting also, i wouldn’t normally ask this, but you said you are new to swift, so are you compiling in production mode? (
the construction returned by the init(from json:JSON) throws
{
(self.timestamp, self.message, self.logger, self.thread, self.level) =
try json.lint
{
(
timestamp: try $0.remove("@timestamp", as: String.self),
message: try $0.remove("message", as: String.self),
logger: try $0.remove("logger_name", as: String.self),
thread: try $0.remove("thread_name", as: String.self),
level: try $0.remove("level") { try $0.case(of: LoggingLevel.self) },
)
}
} where enum LoggingLevel:String
{
case info = "INFO"
case warn = "WARN"
case debug = "DEBUG"
...
} the easiest way to create the
the README is correct, it shows the next planned release, which is 0.3.0, and does not have a tag yet. “0.3.0” right now simply refers to the current |
Apologies for the delayed reply, been trying to figure out how to make the build work but I've done it now. You can find a sample application at https://github.com/rj93/json-decode You were right though, I also tried the 0.3.0 API, but it didn't make any noticeable improvements Any pointers to how to speed that up in my use case? |
it sounds like you were already compiling in release mode then, since going from debug to release mode should cause a way bigger difference than half a second. based on what you described earlier, the performance problem is most likely not in the decoding stage, so switching to the 0.3.0 API would not have made a big difference, since that wasn’t the bottleneck. based on what you have tried and what you’ve described, this is probably a bug in swift-json. i will try to reproduce it and investigate this further with the example project you linked. thanks for the report! |
hi @rj93 , i was not able to reproduce your results on linux, in fact
the only change i made to the example program you provided was adding a i will investigate adding your test data to a CI workflow, to see if there are any potential platform differences that could be causing the performance discrepancy. |
update: i have added your test case to the project’s CI benchmarks here: https://github.com/kelvin13/swift-json/runs/7586938970?check_suite_focus=true#step:7:64 i was also able to significantly optimize your Foundation
i will keep this issue open for now, as it is possible this only affects your particular macOS/Xcode configuration, and we do not yet have CI workflows set up to run benchmarks on macOS. |
@Kelvin13 I checked out project, updated a few things to make it run on macOS (removed the timing functions and used the
So this can probably be closed as it is pretty evident that Thanks for taking the time to look into it. |
I tried implementing exactly what was in the benchmark code and it's still slower than the foundation implementation, so it must be something that I'm doing which is very wrong from with within my app. |
I have a JSON line file and I'm trying to parse it, I'm wondering if this can be used?
Currently
swift-json
is ~4 times slower than the Foundation implementation, but I suspect this is because I am recreating the decoder each time:swift-json (~97 seconds)
Foundation (~24 seconds)
Tested on a ~100MB file with 140000 lines
The text was updated successfully, but these errors were encountered: