Includes don't work on vbcc

[afalkenhahn]

AmiSSL includes don't work with vbcc either. There are these lines in proto/amissl.h:

 #elif defined(__VBCC__)
  #ifndef __PPC__
   #include <inline/amissl_protos.h>
  #endif /* __PPC__ */

But the file inline/amissl_protos.h isn't there. Same for amisslmaster.

Please fix.

[chris-y]

I did the "obvious" rename of inline/amissl.h => inline/amissl_protos.h (and same for amisslmaster), located a copy of the missing inline/macros.h and... it still doesn't work. These are the errors I get:

>typedef	_BSD_SIZE_T_	size_t;
warning 226 in line 29 of "sys/types.h": redeclaration of typedef <size_t>
>typedef signed char int8_t;
warning 226 in line 4 of "stdint.h": redeclaration of typedef <int8_t>
>typedef signed short int16_t;
warning 226 in line 5 of "stdint.h": redeclaration of typedef <int16_t>
>typedef signed int int32_t;
warning 226 in line 6 of "stdint.h": redeclaration of typedef <int32_t>
>                              unsigned char *pass, ossl_ssize_t passlen)
warning 63 in line 152 of "openssl/cms.h": mixed identifier- and parameter-type-list
>                              unsigned char *pass, ossl_ssize_t passlen)
warning 57 in line 152 of "openssl/cms.h": , expected
>STACK_OF(CMS_RecipientInfo)
warning 126 in line 154 of "openssl/cms.h": no declarator in prototype
>STACK_OF(CMS_RecipientInfo)
warning 126 in line 154 of "openssl/cms.h": no declarator in prototype
>                                    ossl_ssize_t passlen)
warning 63 in line 192 of "openssl/cms.h": mixed identifier- and parameter-type-list
>                                    ossl_ssize_t passlen)
warning 57 in line 192 of "openssl/cms.h": , expected
>CMS_RecipientInfo 
warning 126 in line 194 of "openssl/cms.h": no declarator in prototype
>CMS_RecipientInfo 
warning 126 in line 194 of "openssl/cms.h": no declarator in prototype
>                                               ossl_ssize_t passlen,
warning 63 in line 198 of "openssl/cms.h": mixed identifier- and parameter-type-list
>                                               ossl_ssize_t passlen,
warning 57 in line 198 of "openssl/cms.h": , expected
>                                               const EVP_CIPHER *kekciph)
warning 63 in line 199 of "openssl/cms.h": mixed identifier- and parameter-type-list
>int 
warning 126 in line 201 of "openssl/cms.h": no declarator in prototype
>int 
warning 126 in line 201 of "openssl/cms.h": no declarator in prototype
>DEFINE_STACK_OF(SCT)
error 76 in line 123 of "amissl/inline.h": identifier expected
aborting...
unexpected end of file
1 error found!

[Futaura]

Presuming we are referring to the m68k target, those inline files are for GCC only, so renaming is not going to work. Likewise, looks like you are using a GCC sys/types.h for VBCC - that also won't work.

Inline headers for VBCC are not supplied and as far as I can tell only fd2pragma (in VBCC:bin) can generate then, and using that is further complicated by the prototypes being scattered amongst all the openssl header files and not in a single clib file.

@jens-maus is using sfdc in his build environment, so not sure if fd2pragma is an option (I don't think sfdc can output VBCC headers).

[broadblues]

I built an AmiSSL SDK for VBCC The main file egnerate is the inline/amissl_protos.h built with fd2prgam and alot f pateient updating of the types file plus some post processing with the included perl script to deal with functions that returned a pointer to function. There is also a manual fix to deal with a missligned arguement of type doube.

https://www.dropbox.com/s/2h16tm3pk00vvm3/AmiSSLVBCC.lha?dl=0

[jens-maus]

Sorry, but please use GitHub to communicate/propose your changes via an own fork of the AmiSSL repository and generating a pull request. This would make it way easier to review your changes and comment on them. This is not how development works in 2019...

[chris-y]

@broadblues

I tested your version of the includes, and am still getting errors with DEFINE_STACK_OF. I can't actually see where that is defined so not sure what types it doesn't like.

>DEFINE_STACK_OF(SCT)
error 39 in line 131 of "amissl/inline.h": invalid types for assignment
>DEFINE_STACK_OF(CTLOG)
error 39 in line 132 of "amissl/inline.h": invalid types for assignment
>DEFINE_STACK_OF(X509_ALGOR)
error 39 in line 133 of "amissl/inline.h": invalid types for assignment
>DEFINE_STACK_OF(ASN1_STRING_TABLE)
error 39 in line 134 of "amissl/inline.h": invalid types for assignment
>DEFINE_STACK_OF(ASN1_INTEGER)
error 39 in line 135 of "amissl/inline.h": invalid types for assignment
>DEFINE_STACK_OF(ASN1_GENERALSTRING)
error 39 in line 136 of "amissl/inline.h": invalid types for assignment
>DEFINE_STACK_OF(ASN1_UTF8STRING)
error 39 in line 137 of "amissl/inline.h": invalid types for assignment
>DEFINE_STACK_OF(ASN1_TYPE)
error 39 in line 138 of "amissl/inline.h": invalid types for assignment
>DEFINE_STACK_OF(ASN1_OBJECT)
error 39 in line 139 of "amissl/inline.h": invalid types for assignment
>DEFINE_STACK_OF(CMS_SignerInfo)
error 39 in line 140 of "amissl/inline.h": invalid types for assignment

[broadblues]

@chris-y

Yes. These amissl/inlines, create single long lines of code that make workng out what was wrong is nearly impossible. DEFINE_STACK_OF(sometype) creates a struct stack_st_sometype defintion, I don;t think that itself is the error.

This is why I added the

#if defined(VBCC)
#define AMISSL_INLINE_H
#endif

to the example to prevent those inlines being included.

If we can get this bit workng too that would be great but I don't even know what this code is trying to do.

[Futaura]

@chris-y safestack.h, IIRC.

@jens-maus @broadblues I think the ultimate or only proper solution is to extend https://github.com/adtools/sfdc to provide an option to output VBCC files, so that we can easily generate the VBCC files from the .sfd (which is auto generated from the .xml).

[chris-y]

@broadblues
Nearly builds now but I get an "identifier expected" on CloseAmiSSL();
My code is near identical to the cleanup code in the example so not sure what this is trying to tell me.

[jens-maus]

@Futaura Exactly. I also think sfdc should be extended instead of trying to hack together some VBCC binaries of AmiSSL.

[broadblues]

I haven't 'hacked' anything I just added the type definitions to fd2pragma so that the sfd could be parsed without error.

[broadblues]

I've split the lines of the DEFINE_STACK_OF() macro output in the preprocessed output (-E option) and recompiled.

I see the error ocurs at this line

return ( struct stack_st_SCT * ) __OPENSSL_sk_new_reserve ( AmiSSLBase , ( ( OPENSSL_sk_compfunc ) compare ) , ( n ) ) ; 

The inline for that function is

OPENSSL_STACK * __OPENSSL_sk_new_reserve ( __reg ( "a6" ) struct Library * , __reg ( "d0" ) LONG c , __reg ( "d1" ) LONG n ) = "\tjsr\t-31152(a6)" ;

Basically the pointer to function has been taken as an int by fd2pragma, and vbcc doesn't let the pointer to function be assigned to the LONG argument.

Possibly some more type fixing required.

[Futaura]

@broadblues @jens-maus That's annoying - I knew I'd make a mistake somewhere. That function is new for OpenSSL 1.1.1a and obviously that parameter really should have gone into a0.

That said, maybe this ought to be fixed in VBCC (or maybe there is an existing compiler switch) as due to the sometimes insane amount of parameters in OpenSSL functions, d registers may still need to be used for function parameters elsewhere (I know we are already using d registers to pass other pointers).

[Futaura]

@jens-maus So, it turns out I also made mistakes with the several other new functions that take function pointer parameters. Do you think it would be a good idea to do a quick new release that corrects this, on the basis that nobody has likely used these new functions yet? Additionally, if we can look at and fix #28 too. Alternatively, OpenSSL 1.1.1b was released on Tuesday - if you can do the git magic first step of merging that in, I'll happily go through all the changes and fix the .xml file.

[jens-maus]

@Futaura Yes, we can do a quick fix and release a subsequent 4.4 version if you want. Just give me some time to merge in the OpenSSL 1.1.1b version into our vendor branch so that you can integrate it accordingly.

[broadblues]

I'd did wonder about the D register for a pointer but that isn;t the cause of the error with VBCC it's sinmply that VBCC wont allow assigning a pointer function to a LONG, using a more recent fd2pragma that understands pointer to function types. It generates a correct inline and the code in amissl/inlines.h now compiles.

version appdir:Fd2pragma full
fd2pragma 2.197c (20/06/2018)
by Dirk Stoecker software@dstoecker.de

[Futaura]

@broadblues Yes, I later realised my comments about registers we're kinda irrelevant, although I'd still like to fix those up for consistency.

I still think adapting sfdc is a better way to go. Maintaining fd2pragma.types is a complication we can probably do without - the build process is already complicated enough, IMHO.

[broadblues]

The "insane" number of types in OpenSSL ceratainly makes that tedious at the very least so I don't disagree with the sfdc approach. I'm not familar with it, but can take look at it. VBCC inlines seem relatively simple.

WRT resgister issues. RAND_add takes adouble as argument but the register is specified as just d1 , should that not be a register pair? eg d1/d2 ?

Further for VBCC register pairs must be evenly aligned of I understand what Frank Wille told me, so a manual fix was required to the fd2pragma output.
This is what he said.
[quote]
Two problems:

1. double is 64 bits, so it needs you to specify a register pair.
2. vbcc's register pairs always must be even Dn /odd Dn.

Wenn the documentation of AmiSSL says it expects a double in d1/d2 then this
will only work with gcc. vbcc requires d0/d1 or d2/d3. You can work around
that problem with exchanging the registers before calling the library
function:

void __RAND_add(__reg("a6") struct Library *, __reg("a0") const void * buf,
__reg("d2") LONG num, __reg("d0/d1") double randomness) =
"\texg\td1,d2\n\texg\td0,d1\n\t\tjsr\t-7044(a6)";
[/quote]

[broadblues]

Oh another issue I noticed whilst dealing with the fd2pragma output, there is at least one maybe two vararg functions with no stdarg equivalent. These occur later in the xml file so I assume they are new functions, these will work for OS4 I suppose but not for 68k IIUC.

`<method name="OPENSSL_stderr" result="void *" status="unimplemented">
</method>
<method name="OPENSSL_showfatal" result="void">
<arg name="fmta" type="const char *" m68kreg="a0"/>
<vararg name="dummy" type="long" m68kreg="a1"/>
</method>

<method name="OSSL_STORE_LOADER_set_ctrl" result="int">
  <arg name="loader" type="OSSL_STORE_LOADER *" m68kreg="a0"/>
  <arg name="ctrl_function" type="OSSL_STORE_ctrl_fn" m68kreg="d0"/>
</method>
<method name="OSSL_STORE_ctrl" result="int">
  <arg name="ctx" type="OSSL_STORE_CTX *" m68kreg="a0"/>
  <arg name="cmd" type="int" m68kreg="d0"/>
  <vararg name="dummy" type="long" m68kreg="a1"/>
</method>`

[edit] hope the xml shows correctly this time...

[Futaura]

@broadblues Regarding RAND_add(), this needs further investigation - see 9a6329f. From what I can gather it always was a double in OpenSSL, but presumably it was made a float in AmiSSL when it was 68k only, to bypass the register issues that you mention. @f94sbu then changed this to a double for AmiSSL 3.7 for OS4 only. But, it seems this was changed to a double for all targets starting with AmiSSL 4.0 (perhaps inadvertently due to everything now being auto-generated from the .xml file).

OPENSSL_showfatal() I had wanted to remove for 4.3 as it is internal only really, but IIRC it had to be left in for the OpenSSL tests and/or OpenSSL command, and it is a fairly old function. I wouldn't be worried about that, except it means the gcc m68k inline for that function is completely broken (calls another function). Likewise for OSSL_STORE_ctrl(), except there is an equivalent entry for OSSL_STORE_vctrl(), but it does not directly precede OSSL_STORE_vctrl, hence the inline macro calls the wrong function.

[Futaura]

Looking at RAND_add() some more, having tested with both GCC and SAS/C, they both put the double in d1/d2, even though only d1 is explicitly defined - an unwritten rule, I guess. Back in amissl.xml, idltool only reads the first 2 characters of m68kreg, so it is pointless changing that to "d1/d2". I think in the end we are going to have to patch the output from idltool/sfdc for auto-generating VBCC includes, with the changes that Frank has given. Indeed, I tried with VBCC and it does not like __reg("d1") or __reg("d1/d2") for double parameters.

[broadblues]

I've written an experimental patch to sfdc that seems output a good aproximation of a VBCC inline file.

After manually fixing the RAND_add double issue I see quite a few more 64bit types causeing register issues. I suppose these were masked by fd2pragma's type list still being incomplete (I only created those types needed to prevent parsing errors in the first iteration of the process.)

Typical output from compiling at the current stage;

11.Programing:SDK/Examples/AmiSSL> vc +aos68k -c99 -IPrograming:GG/netinclude/ https.c -o https
>typedef long time_t;
warning 226 in line 6 of "time.h": redeclaration of typedef <time_t>
	included from file "vincludeos3:openssl/asn1.h":16
	included from file "vincludeos3:amissl/amissl.h":21
	included from file "vincludeos3:proto/amissl.h":31
	included from file "https.c":23
>int __ASN1_ENUMERATED_set_int64(__reg("a6") struct Library * , __reg("a0") ASN1
error 217 in line 10371 of "inline/amissl_protos.h": register <d0> used with wrong type
	included from file "vincludeos3:proto/amissl.h":69
	included from file "https.c":23
>int __ASN1_INTEGER_set_int64(__reg("a6") struct Library * , __reg("a0") ASN1_IN
error 217 in line 10380 of "inline/amissl_protos.h": register <d0> used with wrong type
	included from file "vincludeos3:proto/amissl.h":69
	included from file "https.c":23
>int __ASN1_INTEGER_set_uint64(__reg("a6") struct Library * , __reg("a0") ASN1_I
error 217 in line 10383 of "inline/amissl_protos.h": register <d0> used with wrong type
	included from file "vincludeos3:proto/amissl.h":69
	included from file "https.c":23
>int __OPENSSL_init_crypto(__reg("a6") struct Library * , __reg("d0") uint64_t o
error 217 in line 10656 of "inline/amissl_protos.h": register <d0> used with wrong type
	included from file "vincludeos3:proto/amissl.h":69
	included from file "https.c":23
>int __EVP_PBE_scrypt(__reg("a6") struct Library * , __reg("a0") const char * pa
error 217 in line 11031 of "inline/amissl_protos.h": register <d5> used with wrong type
	included from file "vincludeos3:proto/amissl.h":69
	included from file "https.c":23
>X509_ALGOR * __PKCS5_pbe2_set_scrypt(__reg("a6") struct Library * , __reg("a0")
error 217 in line 11289 of "inline/amissl_protos.h": register <d3> used with wrong type
	included from file "vincludeos3:proto/amissl.h":69
	included from file "https.c":23
>int __OPENSSL_init_ssl(__reg("a6") struct Library * , __reg("d0") uint64_t opts
error 217 in line 11478 of "inline/amissl_protos.h": register <d0> used with wrong type
	included from file "vincludeos3:proto/amissl.h":69
	included from file "https.c":23
>SCT * __SCT_new_from_base64(__reg("a6") struct Library * , __reg("d0") unsigned
error 217 in line 11769 of "inline/amissl_protos.h": register <d2> used with wrong type
	included from file "vincludeos3:proto/amissl.h":69
	included from file "https.c":23
>void __SCT_set_timestamp(__reg("a6") struct Library * , __reg("a0") SCT * sct ,
error 217 in line 11802 of "inline/amissl_protos.h": register <d0> used with wrong type
	included from file "vincludeos3:proto/amissl.h":69
	included from file "https.c":23
>int __EVP_PKEY_CTX_ctrl_uint64(__reg("a6") struct Library * , __reg("a0") EVP_P
error 217 in line 13676 of "inline/amissl_protos.h": register <d3> used with wrong type
	included from file "vincludeos3:proto/amissl.h":69
	included from file "https.c":23
Maximum number of errors reached!
unexpected end of file
10 errors found!
vbccm68k failed returncode 20
vbccm68k -quiet -hunkdebug "https.c" -o= "T:t_11_0.asm" -c99 -IPrograming:GG/netinclude/  -O=1 -Ivincludeos3: failed

[Futaura]

@jens-maus Related to this, the cross call stubs (for m68k apps calling the PPC AmiSSL) are all broken when it comes to 64-bit parameters. Of course, that means idltool is broken also in this case, because it does not cater for this.

[broadblues]

OK. I have now managed to extend my Macros68kVBBC.pl module for sfdc to attempt to match 64bit types and correct the single registers (ie d0 -> d0/d1) and add further add the register swapping code if the register pair is odd aligned (eg d1/d2)

Also added support for generateing the VBCC style varargs inlines.

I've successfully compiled and executed the https.c example with these inlines. Though thats is far from an exhaustive test!

WRT to vararg functions the previously mentioned OSSL_STORE_ctrl() and OPENSSL_showfatal() are definitly incorrect as being out of order with their stdarg versions the biases used are wrong.

Also I noticed that the va_list argument is in d1 not in address register as the other va_list args are for the other functions.

[Futaura]

Does anyone know how 64-bit parameters should be declared in the .sfd file?

I'm pondering the next step of action which will likely involve updating idltool. It would be best to use the d0/d1 notation in the .xml to clearly define it as 64-bit - it would be quite easy to mistakenly use a register twice, for example, particularly for typedefs that are not obviously 64-bit.

Actually, thinking back, it could well be that the reason I didn't remove OPENSSL_showfatal() was because it messed up the 68k jump table due to the missing stdargs function. I might be able to fix that in idltool too.

[Futaura]

So, from what I can gather there is no way to explicitly denote a 64-bit register pair in a .sfd or .fd file, unless I'm mistaken. This makes things messier as it is handled implicitly/transparently by SAS/C and GCC, but not VBCC.

[broadblues]

IThere actrua appears to be two subtly different syntaxs

MathIeeeDioubBas fd
##base _MathIeeeDoubBasBase
##bias 30
##public
IEEEDPFix(parm)(d0/d1)
IEEEDPFlt(integer)(d0)
IEEEDPCmp(leftParm,rightParm)(d0/d1/d2/d3)
IEEEDPTst(parm)(d0/d1)
IEEEDPAbs(parm)(d0/d1)
IEEEDPNeg(parm)(d0/d1)
IEEEDPAdd(leftParm,rightParm)(d0/d1/d2/d3)
IEEEDPSub(leftParm,rightParm)(d0/d1/d2/d3)
IEEEDPMul(factor1,factor2)(d0/d1/d2/d3)
IEEEDPDiv(dividend,divisor)(d0/d1/d2/d3)
*--- functions in V33 or higher (Release 1.2) ---
IEEEDPFloor(parm)(d0/d1)
IEEEDPCeil(parm)(d0/d1)
##end

MathIeeeTransBas fd
##base _MathIeeeDoubTransBase
##bias 30
##public
IEEEDPAtan(parm)(d0/d1)
IEEEDPSin(parm)(d0/d1)
IEEEDPCos(parm)(d0/d1)
IEEEDPTan(parm)(d0/d1)
IEEEDPSincos(pf2,parm)(a0,d0/d1)
IEEEDPSinh(parm)(d0/d1)
IEEEDPCosh(parm)(d0/d1)
IEEEDPTanh(parm)(d0/d1)
IEEEDPExp(parm)(d0/d1)
IEEEDPLog(parm)(d0/d1)
IEEEDPPow(exp,arg)(d2/d3,d0/d1)
IEEEDPSqrt(parm)(d0/d1)
IEEEDPTieee(parm)(d0/d1)
IEEEDPFieee(single)(d0)
IEEEDPAsin(parm)(d0/d1)
IEEEDPAcos(parm)(d0/d1)
IEEEDPLog10(parm)(d0/d1)
##end

both these are from the 3.5 NDK

[broadblues]

I nace it doesnt't spring out at first glance in the first the registers are just listed in order with no separation f9or arguments. d0/d1/d2/d3

In the second the args are seprate by commas and the register pairs separated by / d0/d1 etc

[Futaura]

I was reading some docs related to FD files and '/' and ',' are interchangeable - the former denoting that a movem instruction could be used. I have been having trouble getting fd2pragma and sfdc working with SFD files containing d0/d1. Using fd2pragma to convert the FD to SFD shows that that d0-d1 is used instead. However, fd2pragma can't handle this when making GCC inlines, but VBCC inlines come out correctly. And sfdc doesn't like it either.

[broadblues]

Somewhat inconsistant between formats and tools then?

Another little issue over type size , it occured to me that openssl_ssize_t defaults to ssize_t which in GCC is probably 64bit. The VBCC posixlib has this set to just an unsigned long ie a 32bit

Is (as I suspect) the 68k version of amissl using 64bit openssl_ssize_t ? I can add another "64bit type pattern" to my fork of sfdc but it would be better if the registers were defined explictly

My fork of sfdc is here BTW broadblues/sfdc

[jens-maus]

@broadblues Perfect. Looking forward to receiving your pull request once your are finished with your sfdc patches.

[Futaura]

@broadblues A little inconsistent, but I have since found a lot of old interesting m68k stuff in v:aug and the C= sfd tool explicitly mentions using a dash for doubles. So, now that's conclusive, I'll finalize my changes for idltool - I presume you have access to the OS4 SVN, or I can send to a bin.

This will most likely mean that sfdc will also need updating to handle 64-bit register pairs.

ssize_t and ossl_ssize_t are 32-bit on OS4 - hope the same is true on m68k @jens-maus (can't check myself). All 64-bit parameters will be declared as register pairs in the AmiSSL .XML file and hence the SFD too.

[Futaura]

I've now committed my idltool 53.26 changes which has the 64-bit support in there.

[Futaura]

I've also now committed some changes to amissl.xml. Due to the OPENSSL_STORE_vctrl() fix, there was no choice but to break backwards compatibility for the new 1.1.1a interface functions. So, I also took the opportunity to fix up and change the m68k registers for those functions. I've also explicitly declared the registers for 64-bit arguments. For backwards compatibility in the older functions, I could not reorder the D registers to ensure the register pair is always even for VBCC compatibility, but I have done for any relevant new 1.1.1a functions. I will also add a note to the README so that we do not forget this in the future.

Anything I've forgotten? Apart from OPENSSL_showfatal() which I shall sort for 1.1.1b - ideally want to remove it without breaking backwards compatibility.

[broadblues]

Great re IDLtool and yes I was mistaken re ssize_t being 64bit so no need to worry about that one.

[Futaura]

@broadblues I've updated idltool again, to 53.27 - this fixes incorrect handling of disabled (unimplemented) vararg functions in the sfd file generation. This now allows us to completely disable OPENSSL_showfatal() in the public API as I had really wanted to do for AmiSSL 4.3.

@jens-maus It has just occurred to me that I didn't formally accept your earlier offer of merging in the OpenSSL 1.1.1b sources - I'd be grateful if you could please do so when you get a chance - I guess it can even go straight into the master branch. If the changelog is anything to go by, no a lot changed, so hopefully it should be a relatively straightforward update. Note that idltool 53.27 is now a formal minimum requirement and sfdc may need adapting to handle the 64-bit parameters for GCC m68k output. My plan is to make AmiSSL 4.4 backwards compatible with 4.0-4.2, so apps compiled for those versions will automatically use 4.4 without recompiling. Apps compiled to use 4.3 will continue to use 4.3 fine as usual, but will need recompiling in order to use 4.4.

[broadblues]

OK I'll grab that IDLTool update.

BTW I did the pull request for my sfdc changes. Being as it was my first ever use of that technic I may or may not have dome it right :-)

WRT gcc registers I noticed that sfdc explicitly truncates and register pairs passed in from the sfd in the format d0-d1 to just d0. (I switch that off for vbcc only so so not to mess up any existing gcc stuff until it's fixed properly). BTW d0/d1 style causes a silent error where the function prototype somehow ends up as part of the return type for the next function!

[walkero-gr]

Hello everyone. I really appreciate your work on this issue. I would like to know the status of it, if it is completed and will be included in a future version of AmiSSL. I am developing a VBCC development environment based on docker (https://github.com/walkero-gr/docker4AmigaVBCC) and I would like to include the AmiSSL includes in the docker image.

Thank you again for the hard work you do.

[Futaura]

@walkero-gr As noted above, I've now updated to the new sfdc with @broadblues VBCC additions, and have checked in the generated VBCC inlines. Does everything look in order and/or did I miss something? These will make it into AmiSSL 4.4, but the includes are not final yet.