Import of Hipchat export (tar.gz) is failing with invalid format #11355

icerootsvw · 2018-07-05T11:08:33Z

Rocket.Chat Version: 0.65.1 (snap installation)
Host-System: Debian GNU Linux Stretch

I created an export of my current Hipchat Installation (Atlassian HipChat 2.2.9 (2018.01.23.110010)). This is a standalone installation and not the cloud version.

The export was an AES encrypted file (around 950MB)
hipchat-2018-07-05_09-30-12.tar.gz.aes: openssl enc'd data with salted password

I decrypted the file with
openssl aes-256-cbc -d -in /home/mbasse/hipchat-2018-07-05_09-30-12.tar.gz.aes -out /home/mbasse/hipchat-2018-07-05_09-30-12.tar.gz -pass pass:******

So i got the tar.gz file
hipchat-2018-07-05_09-30-12.tar.gz: gzip compressed data, was "hipchat-2018-07-05_09-30-12.tar", last modified: Thu Jul 5 09:30:12 2018, max compression

I was able to open the tar.gz file with my local file browser. I was also able to extract it.

I tried to import the file with these 2 options (because i dont know what the difference between Hipchat and Hipchat Enterprise is
FQDN/admin/import/prepare/hipchat
and also
FQDN/admin/import/prepare/hipchatenterprise

In both cases i got after some seconds the error message "Ungültiges Importformat" (english: invalidf import format)

I hope these information are useful. If you need further information please feel free to ask.

Greetings from Germany
Michael

icerootsvw · 2018-07-05T11:11:56Z

as i just figured out, there is a new hipchat version (2.4.0). I will update the system and will recheck the import

icerootsvw · 2018-07-05T14:44:21Z

even with Hipchat 2.4.0 and the export from that system i am getting the exact same error

piotrkochan · 2018-07-17T08:03:10Z

I'm getting a lot of errors:

I20180717-09:55:06.290(2)? Exception in callback of async function: SyntaxError: Unexpected token , in JSON at position 5
I20180717-09:55:06.290(2)?     at JSON.parse (<anonymous>:null:null)
I20180717-09:55:06.291(2)?     at stream.on.Meteor.bindEnvironment.chunk (/media/sdfsdfsdf/Rocket.Chat/.meteor/local/build/programs/server/packages/rocketchat_importer-hipchat-enterprise.js:125:31)
I20180717-09:55:06.291(2)?     at runWithEnvironment (packages/meteor.js:1238:24)

I've looked into source and debugged packages/rocketchat-importer-hipchat-enterprise/server/importer.js. It is failing on this JSON.parse(chunk):

const promise = new Promise((resolve, reject) => {
			this.extract.on('entry', Meteor.bindEnvironment((header, stream, next) => {
				if (header.name.indexOf('.json') !== -1) {
					const info = this.path.parse(header.name);

					stream.on('data', Meteor.bindEnvironment((chunk) => {
						this.logger.debug(`Processing the file: ${ header.name }`);
						const file = JSON.parse(chunk);

because chunk is invalid JSON like:

 y": "private",       "room_admins": [
    452
      ],
  "topic": ""
    }
  },

Maybe it is problem with 'tar-stream' package.

jakob-lundberg · 2018-07-30T09:18:42Z

It looks like the importer takes chunks out of the tar stream and tries to JSON parse them. But to get correct JSON the entire file is needed at once. Maybe the the chunks first needs to be collected in a temp file and then parsed?

piotrkochan · 2018-07-30T10:36:07Z

But I assume it worked before W dniu pon., 30.07.2018 o 10:19 Jakob Lundberg <notifications@github.com> napisał(a):

…

It looks like the importer takes chunks out of the tar stream and tries to JSON parse them. But to get correct JSON the entire file is needed at once. Maybe the the chunks first needs to be collected in a temp file and then parsed? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#11355 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFXy-UZwNR8ZHmkqkjR3IwJCdIzIBi_Hks5uLs-KgaJpZM4VDsIt> .

piotrkochan · 2018-07-30T10:37:30Z

And loading json into file means it must be loaded into memory after so file step is unnecessary. U think the better way is to use json stream package.

…

But I assume it worked before W dniu pon., 30.07.2018 o 10:19 Jakob Lundberg ***@***.***> napisał(a): > It looks like the importer takes chunks out of the tar stream and tries > to JSON parse them. But to get correct JSON the entire file is needed at > once. Maybe the the chunks first needs to be collected in a temp file and > then parsed? > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > <#11355 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AFXy-UZwNR8ZHmkqkjR3IwJCdIzIBi_Hks5uLs-KgaJpZM4VDsIt> > . >

jakob-lundberg · 2018-07-30T12:57:30Z

Well loading everything into memory is also possible. Here is how it could work
jakob-lundberg@dd770b2

piotrkochan · 2018-08-02T06:12:19Z

Still, this is very memory consuming way. I think that is would be better to use JSON Stream, something in the way like:

this.extract.on('entry', Meteor.bindEnvironment((header, stream, next) => {
[....]
stream
    .pipe(this.JSONStream.parse('*'))
    .on('data', Meteor.bindEnvironment((chunk) => {
        console.log(chunk); // {User: ...} or {Room: ...} etc
    })

In this approach tar-stream streams file contents while Json Strem tries to assemble them on the fly. "header" is still available so could be used to detect which file is being read.

icerootsvw · 2018-08-06T14:48:37Z

Is there something how i can support you (i am not a developer).
Maybe this is interesting for you. Hipchat will be canceled and Attlassian will switch to slack in 02.2019.

https://www.theverge.com/2018/7/26/17619482/slack-hipchat-acquisition-stride-atlassian-partnership-microsoft-teams-competition

So i guess there will be more users in the next months which are migrating from Hipchat Server to Rocket.Chat (like we are doing)

jakob-lundberg · 2018-08-08T19:26:58Z

@piotrkochan Will you make a pull request with your solution? Otherwise I can make one with my simple "load entire file into memory" approach.

gf-raveinid · 2018-09-11T22:21:44Z

Rocket.chat
Version | 0.69.1

I'm still getting invalid import file type error. New export from Atlassian HipChat 2.4.1 (2018.07.19.134050)
[34mI20180911-22:17:43.744(0) rocketchat_logger rocketchat_logger.js:278 �[34mMeteor ➔ method setupImporter -> userId: sebDhFRBN9hdySJja , arguments: [ 'hipchatenterprise' ]
�[34mI20180911-22:17:43.770(0) rocketchat_logger rocketchat_logger.js:278 �[34mMeteor ➔ method getImportProgress -> userId: sebDhFRBN9hdySJja , arguments: [ 'hipchatenterprise' ]
�[34mI20180911-22:18:00.170(0) rocketchat_logger rocketchat_logger.js:278 �[34mMeteor ➔ method prepareImport -> userId: sebDhFRBN9hdySJja , arguments: [ 'hipchatenterprise', '', 'application/gzip', 'hipchat-2018-09-11_21-58-51.tar.gz' ]
�[34mI20180911-22:18:00.174(0) Exception while invoking method 'prepareImport' TypeError: First argument must be a string, Buffer, ArrayBuffer, Array, or array-like object. at Function.Buffer.from (buffer.js:183:11) at new Buffer (buffer.js:158:17) at HipChatEnterpriseImporter.prepare (/app/bundle/programs/server/packages/rocketchat_importer.js:221:41) at HipChatEnterpriseImporter.prepare (/app/bundle/programs/server/packages/rocketchat_importer-hipchat-enterprise.js:139:11) at MethodInvocation.prepareImport (/app/bundle/programs/server/packages/rocketchat_importer.js:812:39) at MethodInvocation.methodsMap.(anonymous function) (/app/bundle/programs/server/packages/rocketchat_lib.js:2496:36) at MethodInvocation.methodMap.(anonymous function) (packages/rocketchat_monitoring.js:2731:30) at maybeAuditArgumentChecks (/app/bundle/programs/server/packages/ddp-server.js:1877:12) at DDP._CurrentMethodInvocation.withValue (/app/bundle/programs/server/packages/ddp-server.js:902:126) at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1186:12) at DDPServer._CurrentWriteFence.withValue (/app/bundle/programs/server/packages/ddp-server.js:902:98) at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1186:12) at Promise (/app/bundle/programs/server/packages/ddp-server.js:902:46) at new Promise (:null:null) at Session.method (/app/bundle/programs/server/packages/ddp-server.js:875:23) at /app/bundle/programs/server/packages/ddp-server.js:754:85

piotrkochan · 2018-09-14T11:14:03Z

@jakob-lundberg sorry I can't

artem-levashov · 2018-09-24T11:17:53Z

i see same error
Exception while invoking method 'prepareImport' TypeError: First argument must be a string, Buffer, ArrayBuffer, Array, or array-like object.

It happens when i try to upload 950Mb file.

But when i try to upload 24Mb then this stage is ok. but then everything stops on following
Sep 24 04:55:38 xloo-rchat rocketchat: rocketchat_logger rocketchat_logger.js:278 #33[34mHipChat (tar.gz) Importer ➔ debug#033[39m HipChat (tar.gz) is now at importer_preparing_started.
Sep 24 04:55:38 xloo-rchat rocketchat: rocketchat_logger rocketchat_logger.js:278 #33[34mHipChat (tar.gz) Importer ➔ debug#033[39m Processing the file: users.json
Sep 24 04:55:38 xloo-rchat rocketchat: rocketchat_logger rocketchat_logger.js:278 #33[34mHipChat (tar.gz) Importer ➔ debug#033[39m HipChat (tar.gz) is now at importer_preparing_users.
Sep 24 04:55:38 xloo-rchat rocketchat: rocketchat_logger rocketchat_logger.js:278 #33[34mHipChat (tar.gz) Importer ➔ debug#033[39m Processing the file: rooms.json
Sep 24 04:55:38 xloo-rchat rocketchat: rocketchat_logger rocketchat_logger.js:278 #33[34mHipChat (tar.gz) Importer ➔ debug#033[39m HipChat (tar.gz) is now at importer_preparing_channels.
Sep 24 04:55:38 xloo-rchat rocketchat: rocketchat_logger rocketchat_logger.js:278 #33[34mHipChat (tar.gz) Importer ➔ debug#033[39m HipChat (tar.gz) is now at importer_preparing_messages.
Sep 24 04:55:38 xloo-rchat rocketchat: rocketchat_logger rocketchat_logger.js:278 #33[35mHipChat (tar.gz) Importer ➔ warn#033[39m The loaded users count 268, the loaded rooms 217, and the loaded messages 0
Sep 24 04:55:38 xloo-rchat rocketchat: rocketchat_logger rocketchat_logger.js:278 #33[34mHipChat (tar.gz) Importer ➔ debug#033[39m HipChat (tar.gz) is now at importer_import_failed.

My small archive contains only users and rooms and no massages and no attachments

artem-levashov · 2018-09-24T11:59:22Z

i found in the code if (tempUsers.length === 0 || tempRooms.length === 0 || messagesCount === 0) {
... return }

it looks like i canot import without messages. but with messages my file is too large. i will try to copy several messages from be archive to small

piotrkochan · 2018-09-24T12:27:17Z

I recommend upload file WITHOUT ATTACHMENTS. Rocket will download them online pon., 24 wrz 2018 o 14:00 artem-levashov <notifications@github.com> napisał(a):

…

i found in the code if (tempUsers.length === 0 || tempRooms.length === 0 || messagesCount === 0) { ... return } it looks like i canot import without messages. but with messages my file is too large. i will try to copy several messages from be archive to small — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#11355 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFXy-TqWrQF909rDbN0O5KzL2WXkI52-ks5ueMlPgaJpZM4VDsIt> .

artem-levashov · 2018-09-24T12:48:57Z

with history of only 1 room import works and i`m able now to select what i want to import (but unfortunatly here is also problem - Buttons Uncheck deleted and archived are not working - need to unchek each item manualy.)

so here is still big problem for me how to transfer history. my full history wihtout attachments has size 950Mb in archive - it is about 3 years of using hipchat. it looks like Rocket do not wait while such big size will be uploaded. I think a lot of people needs it to work - the February is close :-)

manuelbliemel · 2018-12-11T11:47:04Z

I'm stuck with the same problem.
Even without attachments the import does not work.
Has anybody managed to solve the issue?

marcinkorzycki · 2019-01-09T09:03:09Z

Bump - the same issue here.

piotrkochan · 2019-01-09T16:45:58Z

@marcinkorzycki RocketChat 0.73.0 introduced some Hipchat Enterprise importer improvements (#12985), did You tried it?

jandeger · 2019-01-11T07:37:49Z

I have the same problem when trying to import from Stride. I have a tar,gz file that I can see all users, rooms etc. I can select the file and in the GUI I see all users and rooms but the buttons doesn't work. Please try to fix this ASAP since only a few weeks left before HipChat/Stride is closed and Slack is not an option for us.
I've updated to latest version 0.73.2 but still the same problem.

marcoj · 2019-01-23T10:10:29Z

We're having exactly the same issue. any news on this? The Stride end date is coming close and Rocket is exactly what we need...

artem-levashov · 2019-01-23T10:47:44Z

We was able to move from hipchat to rocket only by longway workaround. We have dowloaded history by small periods. Each file was no more then 100mb maybe less. We spent about 1.5 weeks to upload history of 2 years.

marcoj · 2019-01-23T11:18:02Z

Well, we're trying to import only the chat rooms + users and that's around a 2mb file... so file size is not the issue

jandeger · 2019-01-23T11:31:32Z

In the new version 0.74 that is coming I saw there was something fixed for HipChat import. From the support they told us that they only support import from HipChat not Stride, I think it's the same format of the export file though. Anyway, we have give up and are moving to Slack, quite good offer the first year for the paid plan.

artem-levashov · 2019-01-23T14:16:52Z

Check room topics. We had problem with import when russian letters was used in room name. I had to rename rooms before transfer. Dont forget to chekc deleted rooms also.

artem-levashov · 2019-01-23T14:18:35Z

Antoher problem was with users with same email but different names. Check that you dont have inactive users with same emailn as active

Hudell · 2019-02-18T18:12:36Z

All known hipchat import issues have been fixed with the release of version 0.74.3. If anybody runs into any issue with it, please open a new issue.

piotrkochan mentioned this issue Jul 17, 2018

Hipchat Enterprise Import fails with unexpected token errors #7932

Closed

jeffwaddell mentioned this issue Jul 28, 2018

HipChat Import Failure #11592

Closed

MartinSchoeler added the feat: import label Jul 31, 2018

jakob-lundberg mentioned this issue Aug 13, 2018

[FIX] Load whole file into memory before parsing JSON. #11755

Closed

gf-raveinid mentioned this issue Sep 11, 2018

HipChat import fails for large backup (e.g. 3 GB) #11693

Closed

tassoevan assigned Hudell Feb 18, 2019

Hudell closed this as completed Feb 18, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Import of Hipchat export (tar.gz) is failing with invalid format #11355

Import of Hipchat export (tar.gz) is failing with invalid format #11355

icerootsvw commented Jul 5, 2018

icerootsvw commented Jul 5, 2018

icerootsvw commented Jul 5, 2018

piotrkochan commented Jul 17, 2018

jakob-lundberg commented Jul 30, 2018

piotrkochan commented Jul 30, 2018 via email

piotrkochan commented Jul 30, 2018 via email •

edited

Loading

jakob-lundberg commented Jul 30, 2018

piotrkochan commented Aug 2, 2018

icerootsvw commented Aug 6, 2018

jakob-lundberg commented Aug 8, 2018

gf-raveinid commented Sep 11, 2018

piotrkochan commented Sep 14, 2018

artem-levashov commented Sep 24, 2018 •

edited

Loading

artem-levashov commented Sep 24, 2018

piotrkochan commented Sep 24, 2018 via email

artem-levashov commented Sep 24, 2018

manuelbliemel commented Dec 11, 2018

marcinkorzycki commented Jan 9, 2019

piotrkochan commented Jan 9, 2019

jandeger commented Jan 11, 2019

marcoj commented Jan 23, 2019

artem-levashov commented Jan 23, 2019 •

edited

Loading

marcoj commented Jan 23, 2019

jandeger commented Jan 23, 2019

artem-levashov commented Jan 23, 2019 •

edited

Loading

artem-levashov commented Jan 23, 2019 •

edited

Loading

Hudell commented Feb 18, 2019

Import of Hipchat export (tar.gz) is failing with invalid format #11355

Import of Hipchat export (tar.gz) is failing with invalid format #11355

Comments

icerootsvw commented Jul 5, 2018

icerootsvw commented Jul 5, 2018

icerootsvw commented Jul 5, 2018

piotrkochan commented Jul 17, 2018

jakob-lundberg commented Jul 30, 2018

piotrkochan commented Jul 30, 2018 via email

piotrkochan commented Jul 30, 2018 via email • edited Loading

jakob-lundberg commented Jul 30, 2018

piotrkochan commented Aug 2, 2018

icerootsvw commented Aug 6, 2018

jakob-lundberg commented Aug 8, 2018

gf-raveinid commented Sep 11, 2018

piotrkochan commented Sep 14, 2018

artem-levashov commented Sep 24, 2018 • edited Loading

artem-levashov commented Sep 24, 2018

piotrkochan commented Sep 24, 2018 via email

artem-levashov commented Sep 24, 2018

manuelbliemel commented Dec 11, 2018

marcinkorzycki commented Jan 9, 2019

piotrkochan commented Jan 9, 2019

jandeger commented Jan 11, 2019

marcoj commented Jan 23, 2019

artem-levashov commented Jan 23, 2019 • edited Loading

marcoj commented Jan 23, 2019

jandeger commented Jan 23, 2019

artem-levashov commented Jan 23, 2019 • edited Loading

artem-levashov commented Jan 23, 2019 • edited Loading

Hudell commented Feb 18, 2019

piotrkochan commented Jul 30, 2018 via email •

edited

Loading

artem-levashov commented Sep 24, 2018 •

edited

Loading

artem-levashov commented Jan 23, 2019 •

edited

Loading

artem-levashov commented Jan 23, 2019 •

edited

Loading

artem-levashov commented Jan 23, 2019 •

edited

Loading