Fix/rework wserver (Issue 1133) #1164

TitiFix · 2021-05-24T17:13:50Z

Fix/rework wserver/gwd/gwsetup for Windows from commit a6d35c7
(Unix specific functions are isolated to preserve fork operation)

Solved Issues :

BUG #1133 Gwd se plante après une demi minute
BUG #1131 Bad base name triggers "empty page" error response
BUG #1118 GeneWeb7 CGI Debian 10 - Verbose on every page

Summary :

connections without traffic are ignored (and closed after -conn_tmout timeout) -- windows version
taken into account possible abort / reset connection (many cases)
Server resiliency : exception handling is improved to log errors to a file (gwd-error.txt in wd directory); If possible in response to the faulty request the error is display. In DEBUG mode (not in release version) the backtrace is always activated.
the operation of the windows server is in dual process (default) or single process (-noproc). -noproc works but requires checking that all global variables are in an initial state each time request.ml is called
For Windows, the dual-process mode works by exchanging files for requests (other than miscellaneous files). It is less efficient than the mono process mode.
In the absence of a native syslog server on Windows, a file taking into account the internal syslogs of Geneweb is created (syslog.txt in wd directory). The operation of syslog on Linux is not changed.
The operation of the server on Linux systems is not changed (forks are used). However, some changes affect both Linux / Windows environments:
-- HTTP exchanges is now HTTP/1.1 as well as CGI exchanges (CGI/1.1)
-- wserver is the master of the connections: Flush and close must not be done outside of wserver.
-- Mutil.bench now use stderr (sdtout is reserved for CGI mode)
-- The launch of a 2nd Geneweb server on the same port is now refused
-- HTTP status 404 now generated on missing file if requested file is not present.
-- HTTP status 405 (Not allowed) for unsupported methods (PUT, HEAD, etc ...).
-- add missing HTTP header for the index page in request.ml and HTTP 404 when unknown basename
-- replacement charset = iso-8859-1 by utf-8 in the response headers (not uniform everywhere).
-- Fixed mime types, some were wrong / obsolete (font, etc ...)
-- the -cgi option to force CGI mode is no longer needed (remains for test purposes). CGI mode now depends on the presence of the GATEWAY_INTERFACE env. variable defined by the CGI RFC.
-- General files (.js, .css, ...) are now in public cache; Personal images are in private cache
-- Always close HTTP connection (HTTP close header)
Add missing files(previously in server mode, in 7.00, the search returned an HTML page as if the 1st directory level was a base) :
css/bootstrap.min.css.map
js/bootstrap.bundle.min.js.map
etc/webfonts/fa-regular-400.eot (need only for IE)
etc/webfonts/fa-solid-900.eot (need only for IE)
etc/webfonts/fa-brands-400.eot (need only for IE)
etc/favicon.ico
etc/apple-touch-icon-precomposed.png
For detection of error codes (SysError too imprecise), the use of stdlib is prohibited. Only the unix library is used. wserver now manage buffering (adjustable from 1400 oct to 64K with -buf_size option in debug mode);

Test environnement : 64 bits
OS :

Windows 10 2004 (Home & pro edition)
Debian 10.4 (server & CGI mode)
Mac ( see hgouraud )

Browsers :

Edge (Chromnium) 90.0.818.66,
Chrome 90.0.4430.212 (64 bits),
Firefox 88.0.1,
Firefox 78.10.1 ESR
IE 11 (Windows 10 2004)

Network trafic : check with Wireshark.

synchro

Better handling of sexcolor in menubar (avoids failure when no spouse)

AS_OK: handle pedigree collapse when sosa_filter=on

Invert params in aa is_substr baaccc; add 'in' as equivalent to 'is_ubstr'

Synchro 17 mai

hgouraud · 2021-05-25T08:46:35Z

On my Mac, tested in server and CGI mode

sagotch

Thanks for this PR. I only had a quick look for now but this seems to be good work.

Here is my first (quick) review.

bin/wserver/wserver.ml

bin/gwd/gwd.ml

bin/gwd/gwdLog.ml

sagotch · 2021-05-25T09:17:51Z

bin/gwd/request.ml

@@ -912,7 +916,6 @@ let treat_request =
 #endif
        | _ -> incorrect_request
      end conf base ;
-    Output.flush conf ;


What is wrong with flushing here?

I did not remove all the flushes in the code (several places). Output.flush does nothing more (see gwd.ml line 17). Cleaning will have to be done later. Only gwd and wserver can flush and close connections.

bin/wserver/dune

bin/wserver/wserver.ml

sagotch · 2021-05-25T09:59:44Z

bin/wserver/wserver.ml

-                  loop_write 0
-                end;
-                loop ()
+        exit 0


Do we need explicit exit 0?

I did not modify the original code on this point.
We are in the child process (pid = 0).
exit 0 allows not return to the infinite loop of server (line 641 WServer)
therefore necessary

sagotch · 2021-05-25T09:59:59Z

bin/wserver/wserver.ml

+        exit 0
+      with
+      | Unix.Unix_error (Unix.ECONNRESET, _, _)
+      | Unix.Unix_error (Unix.ECONNABORTED, _, _) -> Unix.close t; exit 0


Same question: do we need exit 0?

Same response (necessary)

bin/wserver/wserver.mli

sagotch · 2021-05-25T10:05:17Z

bin/wserver/wserver.ml

-    loop ()
-  with Unix.Unix_error (Unix.ECONNRESET, _, _) -> ()
-
-let close_connection () =


What is wrong with this, and why is shutdown_noerr better?

close_connection did several things
skip_possible_remaining_chars was a useless function that only read empty stream (caused unnecessary waiting time) . only a shutdown / close sequence is needed

TitiFix · 2021-05-25T18:29:25Z

in plugins/export/plugin_export.ml
close_connection should be removed (like flush)
The reason is that wserver should be the master of connection closings. It is unhealthy to have connection management outside wserver. By removing the flush / close, this will allow to go, later, to a more modern HTTP server.

sagotch · 2021-05-28T13:36:16Z

bin/gwd/gwd.ml

    end
  else geneweb_server ()

 let () =
+#ifdef DEBUG
+  Printexc.record_backtrace true;
+  Sys.enable_runtime_warnings false;


Did you mean true? I picked (and corrected) these two lines here : 53d561e

sagotch · 2021-05-28T13:42:01Z

A lot of things is done in gwdLog. Wouldn't this to commits be good enough? I prefer adding as less code as possible in the core of geneweb.

e01ccd482 gwd: log to stderr as default instead of gwd.syslog file
f177589d0 Made syslog optionnal (not enabled by default).

a2line · 2021-05-28T13:51:35Z

Bonjour. Pas encore testé les logs mais : (rah mauvaise manip)

le serveur fonctionne sous Windows (7 ici) sur le master ;
aucun des divers problèmes qu'on a rencontré les semaines précédentes ne sont présents !

a2line · 2021-05-28T14:05:06Z

Quant au log j'ai le même problème signalé à Titifix hier à savoir que je n'arrive plus à avoir la sortie en console (comportement par défaut à l'époque) : * `-log stderr` écrit un fichier `stderr` ; * `-log 2` écrit un fichier `2` ET un fichier `stderr`. Titifix a fait un commit ce matin pour ça sur sa PR je n'ai pas encore testé vu que j'étais revenu tester le master TitiFix@811dc10 Le ven. 28 mai 2021 à 15:42, Julien Sagot ***@***.***> a écrit :

…

A lot of things is done in gwdLog. Wouldn't this to commits be good enough? I prefer adding as less code as possible in the core of geneweb. e01ccd4 gwd: log to stderr as default instead of gwd.syslog file f177589 Made syslog optionnal (not enabled by default). — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1164 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACO7ZZ3O3MGJ5WEZF2ERPBDTP6MTVANCNFSM45NTHA3Q> .

sagotch · 2021-05-28T14:23:21Z

-log <stderr> should be fine (I mean: -log "<stderr>", or -log '<stderr>' or -log \<stderr\>, I have no idea about how to escape this uner windows)

a2line · 2021-05-28T14:55:19Z

Thanks, -log "<stderr>" works!

Commits 25 et 28 mai

TitiFix · 2021-06-04T18:26:08Z

PR retirée.

sagotch · 2021-06-07T07:44:13Z

Les principaux correctifs on été integrés à la main, mais il reste pas mal de choses à voir sur cette PR.

TitiFix and others added 7 commits April 3, 2021 08:50

Merge pull request #1 from geneweb/master

562fc58

synchro

Merge pull request #3 from geneweb/master

555bf6b

Better handling of sexcolor in menubar (avoids failure when no spouse)

Merge pull request #4 from geneweb/master

b02e171

AS_OK: handle pedigree collapse when sosa_filter=on

Merge pull request #8 from geneweb/master

10a1fae

Invert params in aa is_substr baaccc; add 'in' as equivalent to 'is_ubstr'

Merge pull request #10 from geneweb/master

59c84f4

Synchro 17 mai

corrections wserver, issue 1133

4ca4a8c

corrections wserver, issue 1133

841b37a

TitiFix mentioned this pull request May 24, 2021

[BUG]Gwd se plante après une demi minute #1133

Closed

sagotch reviewed May 25, 2021

View reviewed changes

TitiFix added 2 commits May 28, 2021 08:02

corrections diverses

811dc10

ignore wserver, dune & makefile

282458c

sagotch reviewed May 28, 2021

View reviewed changes

TitiFix added 2 commits May 30, 2021 09:17

Merge pull request #12 from geneweb/master

eae2afa

Commits 25 et 28 mai

Merge branch 'fixws2' into fixws

6f5ce9c

TitiFix closed this Jun 4, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix/rework wserver (Issue 1133) #1164

Fix/rework wserver (Issue 1133) #1164

TitiFix commented May 24, 2021

hgouraud commented May 25, 2021

sagotch left a comment

sagotch May 25, 2021

TitiFix May 25, 2021

sagotch May 25, 2021

TitiFix May 25, 2021

sagotch May 25, 2021

TitiFix May 25, 2021

sagotch May 25, 2021

TitiFix May 25, 2021

TitiFix commented May 25, 2021

sagotch May 28, 2021

sagotch commented May 28, 2021

a2line commented May 28, 2021 •

edited

a2line commented May 28, 2021 via email

sagotch commented May 28, 2021 •

edited

a2line commented May 28, 2021

TitiFix commented Jun 4, 2021

sagotch commented Jun 7, 2021

Fix/rework wserver (Issue 1133) #1164

Fix/rework wserver (Issue 1133) #1164

Conversation

TitiFix commented May 24, 2021

hgouraud commented May 25, 2021

sagotch left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TitiFix commented May 25, 2021

Choose a reason for hiding this comment

sagotch commented May 28, 2021

a2line commented May 28, 2021 • edited

a2line commented May 28, 2021 via email

sagotch commented May 28, 2021 • edited

a2line commented May 28, 2021

TitiFix commented Jun 4, 2021

sagotch commented Jun 7, 2021

a2line commented May 28, 2021 •

edited

sagotch commented May 28, 2021 •

edited