一个新的无法解析的HN样本

[cnfree]

最新master分支的代码 convert 时，cajparser.py 直接抛异常

 caj.seek(self._TOC_END_OFFSET + i * 20)
 OSError: [Errno 22] Invalid argument

self._TOC_END_OFFSET 是负值

对任务型教学法的理论基础与课堂实践的思考.zip

[cnfree]

和其他的NH格式文件对比了一下

self._PAGE_NUMBER_OFFSET = 0x90 是对的，4页

self._TOC_NUMBER_OFFSET = 0x158 就不对了

[HinTak]

It does not seem to have a toc, the page text start as early as 0x128 or around, and there is no images. On linux viewer nothing can be seen (font problems I think)

[HinTak]

And the Linux viewer crashes when one tries to blindly copy the text from this doc.

[HinTak]

So the 20-byte page description struct (which contains info about images) is wrong, and substantially shorter for this doc too.

[HinTak]

Hmm, actually the page info struct is still 20-byte, but it starts at 0xD8. Setting TOC_NUMBER_OFFSET to 0 (which means no TOC) and TOC_END_OFFSET to 0xD8 (which means also the start of page info struct) works to separate the content into 4 pages. The page content has very different text positionimg code from other HN files seen so far though.

[HinTak]

I have updated the code f7fbe48 - now it at least tell you we can't "convert" to pdf since it is just OCR'ed text overlay; the "parse" operation (it is more or less a debug option so far) works, and show you a hexdump with the text embedded, so you can see the text inside at least.

In the longer term I think the "parse" operation could be further improved so that "text-extract" works ("text-extract" is rather a pretty version of parse, and without debug info). But it would be nice if you have more HN file of this type, since this is the odd single one of its own kind.

We have 10 HN-like files so far in total, including this one, and 7 of them are very similar to each other. This one is the only one we see which does not have the pages as images, and contains only scanned /positioned texts.

[HinTak]

This file is closest to the one in #33 (which does not even have a HN header), but that file still have images per page.

[zombie110year]

目前知网貌似可以下载到这篇文献（2003 新疆学报）的 PDF 版本了。希望对你们的解密有所帮助。
对任务型教学法的理论基础与课堂实践的思考_魏梅.pdf

[HinTak]

Thanks @zombie110year . One interesting aspect of the pdf is its creator/producer field:

Creator:        ReaderEx_DIS_C 2.1.0 Build 3968
Producer:       CNKI®ReaderEx(2.1.0 Build 3968)

Both the linux version and the windows version contains these strings:

ReaderEx 2.3
ReaderEx_DIS %s
ReaderEx_DIS_C %s
%%Creator: ReaderEx

There might be some simple API of converting to pdf via ReaderEx.dll / libreaderex_x64.so . (not that we want to depend on it, but it is nice to have an "official" way of converting for comparison).

[lelandyang]

Thanks @zombie110year . One interesting aspect of the pdf is its creator/producer field:

Creator:        ReaderEx_DIS_C 2.1.0 Build 3968
Producer:       CNKI®ReaderEx(2.1.0 Build 3968)

Both the linux version and the windows version contains these strings:

ReaderEx 2.3
ReaderEx_DIS %s
ReaderEx_DIS_C %s
%%Creator: ReaderEx

There might be some simple API of converting to pdf via ReaderEx.dll / libreaderex_x64.so . (not that we want to depend on it, but it is nice to have an "official" way of converting for comparison).

I thought likewise before
According to my analyses, the strings belongs to member functions in class PDFCreator, specifically PDFCreator::Finalize, it's possible that it is a custom PDF generation library, something like img2pdf.

B.T.W. There are slight difference in 2 linux versions of cajviewer: the above strings are not found in readerEx shipped with CAJViewer-x86_64-libc-2.24.AppImage

What an awesome project!

On the first glance on this project, I was really surprised because there is already an easy and official way to convert from caj to pdf, and I think it is more convenient for daily use.

With CNKIExpress, you can just open your caj file first. The file should appear on your Recent panel. Then click Import to import it into your Library. After this, right-click on your file, choose Export to PDF, select the destination, and ... bingo!

The pdf information is the same as @lelandyang mentioned before.

[HinTak]

Whole point of this project is that some prefers not to use the official viewer...

Hi, @HinTak . What I mean by mentioning this is to prove that conversion is done within ReaderEx_x64.dll (or libreaderex_x64.so on Linux). Once the official viewer is installed, one may wish to use its functionality to do conversion without using the original GUI.

The following code should work on Windows (tested with version ReaderEx_DIS_C 2.3.0 Build 3982). Porting this is more or less a boring job. You just replace LoadLibrary with dlopen, etc.

#include <stdio.h>

#include <Windows.h>

struct ParamList
{
	long flag[4];
	char *src;
	char *extname;
	// Function pointers for open, read, seek, tell, eof, and close.
	void *pfnFILE[6];
	char *dest;
	// Function pointers.
	void *pfnoss[4];
};

typedef int (*PCAJFILE_DistillPageEx1)(ParamList p);

int main(int argc, char *argv[])
{
	HMODULE hMod = LoadLibrary("ReaderEx_x64.dll");
	if (hMod)
	{
		printf("[D] hMod = %8p\n", hMod);
		PCAJFILE_DistillPageEx1 CAJFILE_DistillPageEx1 = (PCAJFILE_DistillPageEx1)GetProcAddress(hMod, (LPCSTR)216);
		if (CAJFILE_DistillPageEx1)
		{
			printf("[D] CAJFILE_DistillPageEx1 = %8p\n", (void *)CAJFILE_DistillPageEx1);
			ParamList p;
			ZeroMemory(&p, sizeof p);
			p.flag[0] = 0x78;
			p.flag[3] = 0x26;
			char src[256], extname[256], dest[256];
			printf("[?] Source pathname = ");
			scanf("%s", src);
			p.src = src;
			printf("[?] Source extname = ");
			scanf("%s", extname);
			p.extname = extname;
			printf("[?] Destination pathname = ");
			scanf("%s", dest);
			p.dest = dest;
			int result = CAJFILE_DistillPageEx1(p);
			if (result)
			{
				printf("[I] Success\n");
			}
			else
			{
				printf("[E] CAJFILE_DistillPageEx1 failed (%d)\n", result);
			}
		}
		else
		{
			printf("[E] GetProcAddress failed (%d)\n", GetLastError());
		}
	}
	else
	{
		printf("[E] LoadLibrary failed (%d)\n", GetLastError());
	}
}

I have tried all the samples in your repo. All of them can be converted successfully EXCEPT those with TEB format. I received error 10061 as following.

[D] hMod = 000007FEEE2A0000
[D] CAJFILE_DistillPageEx1 = 000007FEEE2FA418
[?] Source pathname = C:\CAJSamples-master\issue-61\1.teb
[?] Source extname = .teb
[?] Destination pathname = C:\CAJSamples-master\issue-61\1.caj
connect failed with error: 10061
[E] CAJFILE_DistillPageEx1 failed (0)

Maybe a key is needed to convert or even open TEB files.

[HinTak]

Thanks for the sample code! I'll think about porting that over to Linux. That gives us some interesting comparisons.

Hello, @HinTak . I have already helped port the code to Linux. There are slight differences that you may not know, so I show them in the code below.

Macros are used to guarantee max platform and compiler compatibility.

#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

// Cross-platform support for Windows & Linux.
#if !(defined(CAJ2PDF_OS_WINDOWS) || defined(CAJ2PDF_OS_LINUX))
	#if (defined(_WIN32) || defined(_WIN64) || defined(__WIN32__) || defined(__TOS_WIN__) || defined(__WINDOWS__))
		#define CAJ2PDF_OS_WINDOWS
	#endif
	#if (defined(linux) || defined(__linux) || defined(__linux__) || defined(__gnu_linux__))
		#define CAJ2PDF_OS_LINUX
	#endif
#endif
#if !(defined(CAJ2PDF_OS_WINDOWS) || defined(CAJ2PDF_OS_LINUX))
	#error unsupported platform
#endif
#if defined(CAJ2PDF_OS_WINDOWS) && defined(CAJ2PDF_OS_LINUX)
	#error conflicting platform definition
#endif

#if defined(CAJ2PDF_OS_WINDOWS)
	#define WIN32_LEAN_AND_MEAN
	#include <Windows.h>
#elif defined(CAJ2PDF_OS_LINUX)
	#include <dlfcn.h>
#endif

struct Parameter
{
	uint32_t flag[4];
	char *src;
	char *extname;
	// Function pointers for open, read, seek, tell, eof, and close.
	void *pfnFILE[6];
	char *dest;
	// Function pointers.
	void *pfnoss[4];
};

#if defined(CAJ2PDF_OS_LINUX)
// CAJFILE_Init required on Linux.
typedef int (*PCAJFILE_Init)(char const *exeDir, char const **, int, char const *tmpDir);
#endif
typedef int (*PCAJFILE_DistillPageEx1)(struct Parameter *param);

char src[256], extname[256], dest[256];

int main(int argc, char *argv[])
{
#if defined(CAJ2PDF_OS_WINDOWS)
	HMODULE handle = LoadLibrary(TEXT(".\\ReaderEx_x64.dll"));
#elif defined(CAJ2PDF_OS_LINUX)
	void *handle = dlopen("./libreaderex_x64.so", RTLD_LAZY);
#endif
	if (handle)
	{
		printf("[D] handle = %p\n", handle);
#if defined(CAJ2PDF_OS_WINDOWS)
		PCAJFILE_DistillPageEx1 CAJFILE_DistillPageEx1;
		// ReaderEx_DIS_C 2.3.0 Build 3982: ordinal value = 216
		*(void **)(&CAJFILE_DistillPageEx1) = GetProcAddress(handle, (LPCSTR)216);
#elif defined(CAJ2PDF_OS_LINUX)
		// CAJFILE_Init required on Linux.
		PCAJFILE_Init CAJFILE_Init;
		PCAJFILE_DistillPageEx1 CAJFILE_DistillPageEx1;
		*(void **)(&CAJFILE_Init) = dlsym(handle, "CAJFILE_Init");
		if (CAJFILE_Init)
		{
			printf("[D] CAJFILE_Init = %p\n", CAJFILE_Init);
			int result = CAJFILE_Init(".", NULL, 0, "./tmp");
			if (result)
			{
				printf("[E] CAJFILE_Init failed (%d)\n", result);
				goto cleanup;
			}
		}
		else
		{
			printf("[E] dlsym failed (%s)\n", dlerror());
			goto cleanup;
		}
		*(void **)(&CAJFILE_DistillPageEx1) = dlsym(handle, "CAJFILE_DistillPageEx1");
#endif
		if (CAJFILE_DistillPageEx1)
		{
			printf("[D] CAJFILE_DistillPageEx1 = %p\n", CAJFILE_DistillPageEx1);
			struct Parameter param;
			memset(&param, 0, sizeof param);
			param.flag[0] = 0x78;
			param.flag[3] = 0x26;
			printf("[?] Source pathname = ");
			scanf("%s", src);
			param.src = src;
			printf("[?] Source extname = ");
			scanf("%s", extname);
			param.extname = extname;
			printf("[?] Destination pathname = ");
			scanf("%s", dest);
			param.dest = dest;
			int result = CAJFILE_DistillPageEx1(&param);
			if (result)
			{
				printf("[I] Success\n");
			}
			else
			{
				printf("[E] CAJFILE_DistillPageEx1 failed (%d)\n", result);
				goto cleanup;
			}
		}
		else
		{
#if defined(CAJ2PDF_OS_WINDOWS)
			printf("[E] GetProcAddress failed (%d)\n", GetLastError());
#elif defined(CAJ2PDF_OS_LINUX)
			printf("[E] dlsym failed (%s)\n", dlerror());
#endif
			goto cleanup;
		}
	cleanup:
#if defined(CAJ2PDF_OS_WINDOWS)
		// If the function succeeds, the return value is nonzero.
		// If the function fails, the return value is zero.
		if (!FreeLibrary(handle))
		{
			printf("[E] FreeLibrary failed (%d)\n", GetLastError());
		}
#elif defined(CAJ2PDF_OS_LINUX)
		// On success, dlclose() returns 0; on error, it returns a nonzero value.
		if (dlclose(handle))
		{
			printf("[E] dlclose failed (%s)\n", dlerror());
		}
#endif
	}
	else
	{
#if defined(CAJ2PDF_OS_WINDOWS)
		printf("[E] LoadLibrary failed (%d)\n", GetLastError());
#elif defined(CAJ2PDF_OS_LINUX)
		printf("[E] dlopen failed (%s)\n", dlerror());
#endif
	}
}

Compile the code, put the executable into cnkiexpress's directory, and you can do test.

Conversions made on Linux version seem font missing. Further investigation needed.