Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Replace Puppy with standard HTTP client? #160

Open
YamatoSecurity opened this issue May 29, 2024 · 9 comments
Open

Replace Puppy with standard HTTP client? #160

YamatoSecurity opened this issue May 29, 2024 · 9 comments
Assignees
@fukusuket
Labels
enhancement New feature or request

Comments

@YamatoSecurity
Copy link
Collaborator

@fukusuket
What about replacing Puppy with https://nim-lang.org/docs/httpclient.html in order to be able to compile static linux binaries?
@YamatoSecurity YamatoSecurity added the enhancement New feature or request label May 29, 2024
@fukusuket
Copy link
Collaborator

@YamatoSecurity
Yes, I think it's good! My only concern is that a dll is required on Windows, but is that OK?

@YamatoSecurity
Copy link
Collaborator Author

@fukusuket Since we already require separate dlls to be bundled I don't think there is any problem to add one more. As long as it can compile statically on linux and macos.

@fukusuket
Copy link
Collaborator

@YamatoSecurity
I see, that's fine then, I agree with replacing it with HTTP client!

@fukusuket
Copy link
Collaborator

I'll try implementing it!

@fukusuket fukusuket self-assigned this May 31, 2024
@fukusuket fukusuket mentioned this issue May 31, 2024
Closed
@fukusuket
Copy link
Collaborator

@YamatoSecurity
I tried the httpclient.html, but unforunatelly it seems that a similar issue also occurs :( It seems difficult to resolve, I'll put it hold on now.

@fukusuket fukusuket removed their assignment Jun 1, 2024
@YamatoSecurity
Copy link
Collaborator Author

@fukusuket Thanks so much for implementing it. It seems the problem is that libpcre is still being compiled dynamically.
This person figured out how to compile it statically: https://forum.nim-lang.org/t/5160
Also, nimpcre might work: https://github.com/genotrance/nimpcre

@fukusuket
Copy link
Collaborator

@YamatoSecurity
Thank you for information! I tried, but got a compile error :(
It was last committed 4 years ago, so it may not be compatible with recent versions of Nim...?😭

fukusuke@ub:~/takajo$ nimble build -d:release --threads:on --gcc.exe:"musl-gcc" --gcc.linkerexe:"musl-gcc" --passL="-static" -d:nimDebugDlOpen
  Verifying dependencies for takajo@2.5.0
     Info:  Dependency on cligen@>= 1.5 already satisfied
  Verifying dependencies for cligen@1.7.0
     Info:  Dependency on suru@#f6f1e607c585b2bc2f71309996643f0555ff6349 already satisfied
  Verifying dependencies for suru@0.3.1
     Info:  Dependency on termstyle@any version already satisfied
  Verifying dependencies for termstyle@0.1.0
     Info:  Dependency on nancy@any version already satisfied
  Verifying dependencies for nancy@0.1.1
     Info:  Dependency on ansiparse@>= 0.2.0 already satisfied
  Verifying dependencies for ansiparse@0.4.0
     Info:  Dependency on jsony@>= 1.1.5 already satisfied
  Verifying dependencies for jsony@1.1.5
     Info:  Dependency on bearssl@any version already satisfied
  Verifying dependencies for bearssl@0.2.2
     Info:  Dependency on unittest2@any version already satisfied
  Verifying dependencies for unittest2@0.2.2
     Info:  Dependency on nimpcre@any version already satisfied
  Verifying dependencies for nimpcre@0.2.1
     Info:  Dependency on nimterop@>= 0.4.4 already satisfied
  Verifying dependencies for nimterop@0.6.13
     Info:  Dependency on regex@>= 0.15.0 already satisfied
  Verifying dependencies for regex@0.25.0
     Info:  Dependency on unicodedb@>= 0.7.2 already satisfied
  Verifying dependencies for unicodedb@0.12.0
     Info:  Dependency on cligen@>= 1.0.0 already satisfied
  Verifying dependencies for cligen@1.7.0
   Building takajo/takajo using c backend
stack trace: (most recent call last)
nimpcre.nim(7, 11)       nimpcre
assertions.nim(41, 14)   failedAssertImpl
assertions.nim(36, 13)   raiseAssert
fatal.nim(53, 5)         sysFatal
/home/fukusuke/.choosenim/toolchains/nim-2.0.2/lib/system/fatal.nim(53, 5) Error: unhandled exception: nimpcre.nim(7, 5) `false` Error: -d:usePcreHeader is required [AssertionDefect]
       Tip: 36 messages have been suppressed, use --verbose to show them.
nimble.nim(229)          buildFromDir

    Error:  Build failed for the package: takajo
fukusuke@ub:~/takajo$ nimble build -d:release --threads:on --gcc.exe:"musl-gcc" --gcc.linkerexe:"musl-gcc" --passL="-static" -d:nimDebugDlOpen -d:usePcreHeader
  Verifying dependencies for takajo@2.5.0
     Info:  Dependency on cligen@>= 1.5 already satisfied
  Verifying dependencies for cligen@1.7.0
     Info:  Dependency on suru@#f6f1e607c585b2bc2f71309996643f0555ff6349 already satisfied
  Verifying dependencies for suru@0.3.1
     Info:  Dependency on termstyle@any version already satisfied
  Verifying dependencies for termstyle@0.1.0
     Info:  Dependency on nancy@any version already satisfied
  Verifying dependencies for nancy@0.1.1
     Info:  Dependency on ansiparse@>= 0.2.0 already satisfied
  Verifying dependencies for ansiparse@0.4.0
     Info:  Dependency on jsony@>= 1.1.5 already satisfied
  Verifying dependencies for jsony@1.1.5
     Info:  Dependency on bearssl@any version already satisfied
  Verifying dependencies for bearssl@0.2.2
     Info:  Dependency on unittest2@any version already satisfied
  Verifying dependencies for unittest2@0.2.2
     Info:  Dependency on nimpcre@any version already satisfied
  Verifying dependencies for nimpcre@0.2.1
     Info:  Dependency on nimterop@>= 0.4.4 already satisfied
  Verifying dependencies for nimterop@0.6.13
     Info:  Dependency on regex@>= 0.15.0 already satisfied
  Verifying dependencies for regex@0.25.0
     Info:  Dependency on unicodedb@>= 0.7.2 already satisfied
  Verifying dependencies for unicodedb@0.12.0
     Info:  Dependency on cligen@>= 1.0.0 already satisfied
  Verifying dependencies for cligen@1.7.0
   Building takajo/takajo using c backend
/home/fukusuke/.nimble/pkgs2/nimpcre-0.2.1-3377b148d8e5e1c87e45259d2d4ebb1495ba354c/nimpcre.nim(24, 10) template/generic instantiation of `getHeader` from here
/home/fukusuke/.nimble/pkgs2/nimterop-0.6.13-a93246b2ad5531db11e51de7b2d188c42d95576a/nimterop/build/getheader.nim(421, 65) Error: type mismatch
Expression: true == 1
  [1] true: bool
  [2] 1: int literal(1)

Expected one of (first mismatch at [position]):
[1] proc `==`(a, b: NimIdent): bool
[1] proc `==`(a, b: NimNode): bool
[1] proc `==`(a, b: NimSym): bool
[1] proc `==`(err1, err2: OSErrorCode): bool
[1] proc `==`(pkg1, pkg2: ConanPackage): bool
[1] proc `==`(pkg1, pkg2: JBBPackage): bool
[1] proc `==`(x, y: char): bool
[1] proc `==`(x, y: cstring): bool
[1] proc `==`(x, y: float): bool
[1] proc `==`(x, y: float32): bool
[1] proc `==`(x, y: int): bool
[1] proc `==`(x, y: int16): bool
[1] proc `==`(x, y: int32): bool
[1] proc `==`(x, y: int64): bool
[1] proc `==`(x, y: int8): bool
[1] proc `==`(x, y: pointer): bool
[1] proc `==`(x, y: string): bool
[1] proc `==`(x, y: uint): bool
[1] proc `==`(x, y: uint16): bool
[1] proc `==`(x, y: uint32): bool
[1] proc `==`(x, y: uint64): bool
[1] proc `==`(x, y: uint8): bool
[1] proc `==`[A, B](s, t: OrderedTableRef[A, B]): bool
[1] proc `==`[A, B](s, t: OrderedTable[A, B]): bool
[1] proc `==`[A, B](s, t: TableRef[A, B]): bool
[1] proc `==`[A, B](s, t: Table[A, B]): bool
[1] proc `==`[A](s, t: CountTableRef[A]): bool
[1] proc `==`[A](s, t: CountTable[A]): bool
[1] proc `==`[Enum: enum](x, y: Enum): bool
[1] proc `==`[I, T](x, y: array[I, T]): bool
[1] proc `==`[T: proc | iterator](x, y: T): bool
[1] proc `==`[T: tuple | object](x, y: T): bool
[1] proc `==`[T](x, y: openArray[T]): bool
[1] proc `==`[T](x, y: ptr T): bool
[1] proc `==`[T](x, y: ref T): bool
[1] proc `==`[T](x, y: seq[T]): bool
[1] proc `==`[T](x, y: set[T]): bool
[2] proc `==`(x, y: bool): bool

       Tip: 28 messages have been suppressed, use --verbose to show them.
nimble.nim(229)          buildFromDir

    Error:  Build failed for the package: takajo

@YamatoSecurity YamatoSecurity mentioned this issue Sep 7, 2024
Closed
@fukusuket
Copy link
Collaborator

fukusuket commented Sep 9, 2024 •
edited
Loading

I was able to solve the pcre compile error, but now I have a sqlite compile error ... :(

git clone https://github.com/kaushalmodi/hello_musl.git
cd hello_musl/
nim musl -d:pcre src/hello_musl_pcre.nim
cd ../
git clone https://github.com/Yamato-Security/takajo.git
cd takajo
git checkout 160-replace-puppy-httpclient
cp ../hello_musl/pcre/8.42/include/pcre.h /usr/include/x86_64-linux-musl/
nimble build -d:release --threads:on -d:nimDebugDlOpen --gcc.exe:musl-gcc --gcc.linkerexe:musl-gcc --passL=-static --passL:"../hello_musl/pcre-8.42/.libs/libpcre.a" -d:usePcreHeader
./takajo
Dynamic loading not supported
Dynamic loading not supported
could not load: libsqlite3.so(|.0)

memo:

sudo apt install libsqlite3-dev

@fukusuket
Copy link
Collaborator

sqlite static compile error memo

wget https://www.sqlite.org/2024/sqlite-autoconf-3460100.tar.gz
tar zxvf sqlite-autoconf-3460100.tar.gz
cd sqlite-autoconf-3460100
CFLAGS="-O2" LDFLAGS="-static" ./configure
make
ls .libs/libsqlite3.a

musl-gcc  -O2           \
  shell.c sqlite3.c     \
    -static -lm         \
    -o sqlite3

nimble build -d:release --threads:on -d:nimDebugDlOpen --gcc.exe:musl-gcc --gcc.linkerexe:musl-gcc --passL=-static --passL:"../hello_musl/pcre-8.42/.libs/libpcre.a" -d:usePcreHeader --dynlibOverride:sqlite3 --passL:"../sqlite-autoconf-3460100/.libs/libsqlite3.a"
$ nimble build -d:release --threads:on -d:nimDebugDlOpen --gcc.exe:musl-gcc --gcc.linkerexe:musl-gcc --passL=-static --passL:"../hello_musl/pcre-8.42/.libs/libpcre.a" -d:usePcreHeader --dynlibOverride:sqlite3 --passL:"../sqlite-autoconf-3460100/.libs/libsqlite3.a" --mm:refc -d:nimDebugDlOpen --verbose
     Info:  Nimble data file "/home/fukusuke/.nimble/nimbledata2.json" has been loaded.
  Verifying dependencies for takajo@2.6.0
    Reading official package list
   Checking for cligen@>= 1.5
     Info:  Dependency on cligen@>= 1.5 already satisfied
  Verifying dependencies for cligen@1.7.0
    Reading official package list
   Checking for suru@#f6f1e607c585b2bc2f71309996643f0555ff6349
     Info:  Dependency on suru@#f6f1e607c585b2bc2f71309996643f0555ff6349 already satisfied
  Verifying dependencies for suru@0.3.1
    Reading official package list
   Checking for termstyle@any version
     Info:  Dependency on termstyle@any version already satisfied
  Verifying dependencies for termstyle@0.1.0
    Reading official package list
   Checking for nancy@any version
     Info:  Dependency on nancy@any version already satisfied
  Verifying dependencies for nancy@0.1.1
    Reading official package list
   Checking for ansiparse@>= 0.2.0
     Info:  Dependency on ansiparse@>= 0.2.0 already satisfied
  Verifying dependencies for ansiparse@0.4.0
    Reading official package list
   Checking for jsony@>= 1.1.5
     Info:  Dependency on jsony@>= 1.1.5 already satisfied
  Verifying dependencies for jsony@1.1.5
    Reading official package list
   Checking for db_connector@>= 0.1.0
     Info:  Dependency on db_connector@>= 0.1.0 already satisfied
  Verifying dependencies for db_connector@0.1.0
    Reading official package list
   Checking for bearssl@any version
     Info:  Dependency on bearssl@any version already satisfied
  Verifying dependencies for bearssl@0.2.2
    Reading official package list
   Checking for unittest2@any version
     Info:  Dependency on unittest2@any version already satisfied
  Verifying dependencies for unittest2@0.2.2
   Building takajo/takajo using c backend
  Executing /home/fukusuke/.nimble/bin/nim c --colors:on --noNimblePath -d:release --threads:on -d:nimDebugDlOpen --gcc.exe:musl-gcc --gcc.linkerexe:musl-gcc --passL:-static --passL:../hello_musl/pcre-8.42/.libs/libpcre.a -d:usePcreHeader --dynlibOverride:sqlite3 --passL:../sqlite-autoconf-3460100/.libs/libsqlite3.a --mm:refc -d:nimDebugDlOpen -d:NimblePkgVersion=2.6.0 --path:/home/fukusuke/.nimble/pkgs2/termstyle-0.1.0-4641c9f9e587d5cf04ccd3ce0bf1ef0263a16299 --path:/home/fukusuke/.nimble/pkgs2/cligen-1.7.0-300bd7fdb6e48d2d98e34ed0661206b50331e99c --path:/home/fukusuke/.nimble/pkgs2/nancy-0.1.1-626d5a8937046ed31841d8af728e98013c476768 --path:/home/fukusuke/.nimble/pkgs2/jsony-1.1.5-6aeb83e7481ca8686396a568096054bc668294df --path:/home/fukusuke/.nimble/pkgs2/suru-0.3.1-06ecd8dfb1294b5469d919c83e14db111c1346be --path:/home/fukusuke/.nimble/pkgs2/unittest2-0.2.2-6ac7f7ea74df812a25be5d6d36a59d23789eb508 --path:/home/fukusuke/.nimble/pkgs2/ansiparse-0.4.0-49411d0e4b4e90df4020386b6f14be8c54809a63 --path:/home/fukusuke/.nimble/pkgs2/db_connector-0.1.0-d68319e3785fa937f0465ea915e942b61b6b5442 --path:/home/fukusuke/.nimble/pkgs2/bearssl-0.2.2-d238564ab1e29087395561368505753e4c25ccde -o:/home/fukusuke/takajo/takajo /home/fukusuke/takajo/src/takajo.nim
Hint: used config file '/home/fukusuke/.choosenim/toolchains/nim-2.0.8/config/nim.cfg' [Conf]
Hint: used config file '/home/fukusuke/.choosenim/toolchains/nim-2.0.8/config/config.nims' [Conf]
Hint: used config file '/home/fukusuke/takajo/config.nims' [Conf]
...............................................................................................................................................................................................................................
/home/fukusuke/takajo/src/takajo.nim(16, 8) Warning: use the nimble packages `malebolgia`, `taskpools` or `weave` instead; threadpool is deprecated [Deprecated]
..........
/home/fukusuke/takajo/src/takajopkg/general.nim(452, 17) Hint: 'x' is declared but not used [XDeclaredButNotUsed]
/home/fukusuke/takajo/src/takajopkg/general.nim(437, 9) Hint: 'length' is declared but not used [XDeclaredButNotUsed]
.................
/home/fukusuke/takajo/src/takajopkg/htmlReport.nim(6, 8) Hint: duplicate import of 'os'; previous import here: /home/fukusuke/takajo/src/takajo.nim(18, 8) [DuplicateModuleImport]
/home/fukusuke/takajo/src/takajopkg/htmlReport.nim(6, 12) Hint: duplicate import of 'strutils'; previous import here: /home/fukusuke/takajo/src/takajo.nim(11, 8) [DuplicateModuleImport]
/home/fukusuke/takajo/src/takajopkg/htmlReport.nim(188, 42) Hint: 'e' is declared but not used [XDeclaredButNotUsed]
..
/home/fukusuke/takajo/src/takajopkg/splitJsonTimeline.nim(33, 13) Hint: 'inputFile' is declared but not used [XDeclaredButNotUsed]
.
/home/fukusuke/takajo/src/takajopkg/stackCmdlines.nim(17, 18) Hint: 'otherColumn' is declared but not used [XDeclaredButNotUsed]
.
/home/fukusuke/takajo/src/takajopkg/stackComputers.nim(18, 18) Hint: 'otherColumn' is declared but not used [XDeclaredButNotUsed]
..
/home/fukusuke/takajo/src/takajopkg/stackIpAddresses.nim(18, 18) Hint: 'otherColumn' is declared but not used [XDeclaredButNotUsed]
..
/home/fukusuke/takajo/src/takajopkg/stackProcesses.nim(17, 18) Hint: 'otherColumn' is declared but not used [XDeclaredButNotUsed]
...
/home/fukusuke/takajo/src/takajopkg/stackUsers.nim(48, 18) Hint: 'otherColumn' is declared but not used [XDeclaredButNotUsed]
..
/home/fukusuke/takajo/src/takajopkg/sysmonProcessTree.nim(54, 13) Hint: 's' is declared but not used [XDeclaredButNotUsed]
/home/fukusuke/takajo/src/takajopkg/sysmonProcessTree.nim(124, 17) Hint: 'timeStamp' is declared but not used [XDeclaredButNotUsed]
.
/home/fukusuke/takajo/src/takajopkg/timelineLogon.nim(171, 17) Hint: 'eventId' is declared but not used [XDeclaredButNotUsed]
..
/home/fukusuke/takajo/src/takajopkg/timelineSuspiciousProcesses.nim(20, 14) Hint: 'pidInt' is declared but not used [XDeclaredButNotUsed]
/home/fukusuke/takajo/src/takajopkg/timelineSuspiciousProcesses.nim(20, 5) Hint: 'eventId' is declared but not used [XDeclaredButNotUsed]
/home/fukusuke/takajo/src/takajopkg/timelineSuspiciousProcesses.nim(21, 5) Hint: 'channel' is declared but not used [XDeclaredButNotUsed]
....
/home/fukusuke/takajo/src/takajopkg/ttpVisualizeSigma.nim(45, 9) Hint: 'x' is declared but not used [XDeclaredButNotUsed]
.
/home/fukusuke/takajo/src/takajopkg/vtDomainLookup.nim(12, 9) Hint: 'malicious' is declared but not used [XDeclaredButNotUsed]
.
/home/fukusuke/takajo/src/takajopkg/vtIpLookup.nim(9, 9) Hint: 'malicious' is declared but not used [XDeclaredButNotUsed]
.
/home/fukusuke/takajo/src/takajopkg/vtHashLookup.nim(11, 9) Hint: 'malicious' is declared but not used [XDeclaredButNotUsed]
.
Hint:  [Link]
/usr/bin/ld: ../sqlite-autoconf-3460100/.libs/libsqlite3.a(sqlite3.o): in function `fts5TriTokenize':
sqlite3.c:(.text+0xc041): undefined reference to `__memmove_chk'
/usr/bin/ld: ../sqlite-autoconf-3460100/.libs/libsqlite3.a(sqlite3.o): in function `sqlite3VdbeMemStringify':
sqlite3.c:(.text+0x307d2): undefined reference to `__memcpy_chk'
/usr/bin/ld: ../sqlite-autoconf-3460100/.libs/libsqlite3.a(sqlite3.o): in function `fts5MergePrefixLists':
sqlite3.c:(.text+0x69bab): undefined reference to `__memset_chk'
/usr/bin/ld: ../sqlite-autoconf-3460100/.libs/libsqlite3.a(sqlite3.o): in function `balance_nonroot':
sqlite3.c:(.text+0x7c110): undefined reference to `__memset_chk'
/usr/bin/ld: sqlite3.c:(.text+0x7c258): undefined reference to `__memset_chk'
/usr/bin/ld: ../sqlite-autoconf-3460100/.libs/libsqlite3.a(sqlite3.o): in function `unixOpen':
sqlite3.c:(.text+0x8a5e5): undefined reference to `__memcpy_chk'
/usr/bin/ld: ../sqlite-autoconf-3460100/.libs/libsqlite3.a(sqlite3.o): in function `whereLoopAddOr':
sqlite3.c:(.text+0x9aea3): undefined reference to `__memcpy_chk'
/usr/bin/ld: sqlite3.c:(.text+0x9af8a): undefined reference to `__memcpy_chk'
/usr/bin/ld: ../sqlite-autoconf-3460100/.libs/libsqlite3.a(sqlite3.o):(.data.rel+0x1f10): undefined reference to `fcntl64'
collect2: error: ld returned 1 exit status
Error: execution of an external program failed: 'musl-gcc  @takajo_linkerArgs.txt'
nimble.nim(304)          buildFromDir

    Error:  Build failed for the package: takajo
     Info:  Nimble data file "/home/fukusuke/.nimble/nimbledata2.json" has been saved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@fukusuket fukusuket

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Replace Puppy with standard HTTP client Yamato-Security/takajo
2 participants
@fukusuket @YamatoSecurity