New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Doesn't work for large files #1

Open
saschalalala opened this Issue Nov 1, 2015 · 3 comments

Comments

Projects
None yet
4 participants
@saschalalala

saschalalala commented Nov 1, 2015

Hey, I tried processing a quite large file (476MB) and get an allocation size overflow.

@evitolins

This comment has been minimized.

Show comment
Hide comment
@evitolins

evitolins Mar 26, 2017

Having the same issue in FF 52.0.1 with a 314.8 MB .json file

evitolins commented Mar 26, 2017

Having the same issue in FF 52.0.1 with a 314.8 MB .json file

@faddat

This comment has been minimized.

Show comment
Hide comment
@faddat

faddat Apr 29, 2017

500MB file dying here as well.

faddat commented Apr 29, 2017

500MB file dying here as well.

@danielhickman

This comment has been minimized.

Show comment
Hide comment
@danielhickman

danielhickman Jun 30, 2017

Maybe someone submitting a PR with an implementation of something like this? I'm not sure if it is similar but I really only wanted a single conversation so I wrote a Ruby script to split my JSON file into many by ID. It messily made ~50 files and you'll need to search the original file to figure out which ID you need. You can also one by one add them to Hangouts Reader them since it doesn't destroy previous parses.

require 'json'

def getJSON(file)
	if File.readable?(file)
		$data = JSON.parse(IO.read(file))["conversation_state"]
	end
end


getJSON("Hangouts.json")
puts "Parsed File"

$restructured = {}
$data.each do |i|
	id = i["conversation_id"]["id"]
	puts "Found conversation in #{id}"
	if !$restructured[id]
		$restructured[id] = {"conversation_state" => []}
	end
	$restructured[id]["conversation_state"] << i
end
puts "Finished sorting"

$restructured.each do |key, value|
	puts "Generating #{key}"
	output = File.new("#{key}.json", "w+")
	output.write(JSON.generate(value))
end
puts "Done"

I imported the 10mb file just fine, but results may vary if you have a really long chat you'd like to import.

danielhickman commented Jun 30, 2017

Maybe someone submitting a PR with an implementation of something like this? I'm not sure if it is similar but I really only wanted a single conversation so I wrote a Ruby script to split my JSON file into many by ID. It messily made ~50 files and you'll need to search the original file to figure out which ID you need. You can also one by one add them to Hangouts Reader them since it doesn't destroy previous parses.

require 'json'

def getJSON(file)
	if File.readable?(file)
		$data = JSON.parse(IO.read(file))["conversation_state"]
	end
end


getJSON("Hangouts.json")
puts "Parsed File"

$restructured = {}
$data.each do |i|
	id = i["conversation_id"]["id"]
	puts "Found conversation in #{id}"
	if !$restructured[id]
		$restructured[id] = {"conversation_state" => []}
	end
	$restructured[id]["conversation_state"] << i
end
puts "Finished sorting"

$restructured.each do |key, value|
	puts "Generating #{key}"
	output = File.new("#{key}.json", "w+")
	output.write(JSON.generate(value))
end
puts "Done"

I imported the 10mb file just fine, but results may vary if you have a really long chat you'd like to import.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment