Missing custom column data in backup file

[ankitsoin]

I am sorry if I am not clear enough in my other thread, thought to create separately as its separate issue.
#224 (reply in thread)

Actually as you suggested before to upgrade to 6.1 version to have union support, but I dont want to lose my data in existing streams. So i thought of taking backup of data as per documentation.
https://kb.timebase.info/community/development/howto#backup-stream-data

Getting error running "export path.stream1.qsmsg"
Java.lang.ClassNotFoundException: A.B.C.MessageClass

When i checked my backup file i.e stream1.qsmsg file it is filled with only symbol column rest all of custom columns data is empty.
May i know why this is happening?

In this way i will never able to backup my whole stream data for future upgrades.
Please help this is quite critical for me.

More over i have created stream same as mentioned in below thread
#225 (comment)

[ankitsoin]

Hello, please help me knowing the solution to backup data

[ankitsoin]

Ok thanks 👍 I will check once.

May I know if I have to do incremental backup how it is possible with and without spaces?
When i use export comand to backup into abc.qsmsg it backs up whole stream in 1 go. But I want to backup my data daily(only for current day not whole stream data).

How I will be able to perform this task?

[alex-karpovich]

you can set start/end timestamps for backup, also list of symbols/types
check 'Replication' section of the TimeBase shell help

[ankitsoin]

Is it possible to do back based on custom column?

Example let say i have different asset type for a day (a, b and c) each containing millions of records i want to backup based upon this asset type lets say 'a' in single backup file and 'b' in separate backup file and same for 'c' .

[ankitsoin]

0c05cd6 added additional command "export-with-spaces" into TimeBase shell @ankitsoin

@alex-karpovich
Is "export-with-spaces" is available in version 6.1.13 that you released recently?
Also, can you please tell me the version for which it is previously available before 6.1.13?

[alex-karpovich]

"export-with-spaces" available in 6.1.13

using with gradlew
implementation "org.finos.timebase-ce:timebase-client:6.1.13"

[alex-karpovich]

we have option to export QQL in TimeBase Administrator

but TimeBase Shell can only export streams for now

[ankitsoin]

HI Alex,

I have tried creating backup using below but after upgrading TimeBase version data is not getting restored from same path.
Please help us in backup/restoring data with spaces.

export-with-spaces dirA/subdir/exp.qsmsg.gz

open rw
set src dirA/subdir/exp.qsmsg.gz
set stream market_volume
import

[ankitsoin]

HI @alex-karpovich, is this requires a fix? as its getting hard to reload whole set of data after every version upgrade.

[alex-karpovich]

@ankitsoin can please confirm that when you restart TimeBase Server, no active writers exists?
you need to stop all writing activities before restart.

[ankitsoin]

I believe, there was no existing write activity at that time.
I can check next time same, if version upgrades.

But is this the root cause to which I am unable to back and restore?
I can see my backup .zip files got generated but getting error while importing.

Does it mean I cannot import now?

[alex-karpovich]

@ankitsoin please use latest version 6.1.20

[ankitsoin]

Hi @alex-karpovich ,

Today we have upgraded TB and confirms that there is no active writes at the time of upgrade.
It is giving same error while restoring as mentioned in last screenshot.

Our backup/restore is working from TB admin manually, but from shell it isn't working.
We want to automate the process of backup and restore using shell.

Can you please help.

[alex-karpovich]

@ankitsoin you can import and export spaces data from latest TimeBase Webadmin https://github.com/epam/TimebaseWS/releases/tag/1.1.2

[ankitsoin]

Hi @alex-karpovich

We are using same version1.1.2.

I clicked on space name under the stream
Then I clicked on export option (top right corner of the data window) and able to export using "FILE BY SPACE" option under config(Export to: one file) then able to download data in zip then unzipped to get qsmsg file.

Importing to the stream:
I choose the file from my local QSMSG which I got while exporting above.
Write Mode INSERT

Selected Periodicity Irregular (from static, regular and irregular)
and clicked on import.

Data get loaded to stream but it got loaded to the "root" folder under the stream.

Can you please help me knowing how to upload it to specific space ? (I cant see any spaces option while importing)

[ankitsoin]

Hi @alex-karpovich,
May I know is this requires a fix or I am missing something?

As it is not even getting imported using TB Shell.

[alex-karpovich]

@ankitsoin - we have no option to upload a specific space from zip.
you need to repack zip to leave required spaces in it.

[ankitsoin]

Hi @alex-karpovich ,

I can able to export/import using full zip(including all spaces) of stream successfully.

Also, I have kept the specific spaces data file into the zip. Imported working fine.
Deletion of spaces has to be done directly by clicking on the zip.