New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

doxygen crashes no resolved (Origin: bugzilla #764515) #5978

Closed
doxygen opened this Issue Jul 2, 2018 · 0 comments

Comments

Projects
None yet
1 participant
@doxygen
Owner

doxygen commented Jul 2, 2018

status RESOLVED severity blocker in component general for ---
Reported in version unspecified on platform Other
Assigned to: Dimitri van Heesch

On 2016-04-02 20:52:54 +0000, Kemin Zhou wrote:

When running the doxygen command, it crashed with version 1.8.10, 1.8.11
but not with 1.61
I have a mixture of C++ and Java files. It crashed on one of the Java files
This is reproducible on different platforms. So it is doxygen specific.

Parsing file /path/to/my/source/LoadCDSFromAce.java...
Segmentation fault: 11

I looked into this file and could not find anything special. This is an really old piece of code that has always been able to pass doxygen without any problem.

On 2016-04-03 04:35:26 +0000, Kemin Zhou wrote:

The directory is mostly C++ file. The first Java files are fine, the third one got segmentation fault. I went through every single line of this crashed file. Everything is normal. I cannot see any abnormal thing in this file.

On 2016-04-03 07:50:40 +0000, albert wrote:

Can you build doxygen yourself in debug mode and see with gdb where it crashes otherwise can you please attach a self-contained example (source+config file in a tar or zip) that allows us to reproduce the problem?

On 2016-04-03 18:21:15 +0000, Kemin Zhou wrote:

I was planning to do debug run because it is very difficult to reproduce on your end. Even on my end it may be a special combination of things. I never had any trouble with doxygen before.

On 2016-04-03 19:32:46 +0000, Kemin Zhou wrote:

Parsing file /home/myhome/proj/ortho/LoadCDSFromAce.java...

Program received signal SIGSEGV, Segmentation fault.
0x0000000000683852 in QCString::StringRep::length (this=0x50)
at /home/myhome/Downloads/doxygen-1.8.11/qtools/qcstring.h:581
581 uint l = u.s.isShort ? u.s.len : u.l.d->len;
(gdb) bt

0 0x0000000000683852 in QCString::StringRep::length (this=0x50)

at /home/myhome/Downloads/doxygen-1.8.11/qtools/qcstring.h:581

1 0x0000000000683002 in QCString::length (this=0x50)

at /home/myhome/Downloads/doxygen-1.8.11/qtools/qcstring.h:197

2 0x000000000068317c in QCString::operator+= (this=0x50, str=0x18d6bbb " ")

at /home/myhome/Downloads/doxygen-1.8.11/qtools/qcstring.h:309

3 0x000000000088855d in scannerYYlex () at scanner.l:2052

4 0x00000000008a5944 in parseMain (

fileName=0x2474f58 "/home/myhome//proj/ortho/LoadCDSFromAce.java", 
fileBuf=0x23692f0 "import acedb.*;\nimport java.util.*;\nimport java.sql.*;\nimport java.io.*;\nimport DBinfo;\n\n/**\n * this class loads both species and CDS table from the component\n * Acedbs to the single Postgres DB\n * sp"..., rt=0x1566190, sameTranslationUnit=false, 
filesInSameTranslationUnit=...) at scanner.l:6922

5 0x00000000008a5d50 in CLanguageScanner::parseInput (this=0x1454710,

fileName=0x2474f58 "/home/myhome/proj/ortho/LoadCDSFromAce.java", 
fileBuf=0x23692f0 "import acedb.*;\nimport java.util.*;\nimport java.sql.*;\nimport java.io.*;\nimport DBinfo;\n\n/**\n * this class loads both species and CDS table from the component\n * Acedbs to the single Postgres DB\n * sp"..., root=0x1566190, sameTranslationUnit=false, 
filesInSameTranslationUnit=...) at scanner.l:7051

6 0x0000000000677876 in parseFile (parser=0x1454710, root=0x1566190, rootNav=0x178fbb0, fd=

0x15b3b20, fn=0x1592938 "/home/myhome/proj/ortho/LoadCDSFromAce.java", 
sameTu=false, filesInSameTu=...)
at /home/myhome/Downloads/doxygen-1.8.11/src/doxygen.cpp:9378

7 0x0000000000677ab2 in parseFiles (root=0x1566190, rootNav=0x178fbb0)

at /home/myhome/Downloads/doxygen-1.8.11/src/doxygen.cpp:9470

8 0x00000000006804d7 in parseInput ()

at /home/myhome/Downloads/doxygen-1.8.11/src/doxygen.cpp:11093

9 0x000000000064a121 in main (argc=2, argv=0x7fffffffe4a8)

at /home/myhome/Downloads/doxygen-1.8.11/src/main.cpp:37

================= the content of the really old and useless java file: LoadCDSFromAce.java ==============
import acedb.;
import java.util.
;
import java.sql.;
import java.io.
;
import DBinfo;

/**

  • this class loads both species and CDS table from the component

  • Acedbs to the single Postgres DB

  • species must be loaded first because cds source is a foreign key
    /
    public class LoadCDSFromAce extends DBinfo {
    private Ace aceDB = null;
    /
    *

    • postgres DB connection
      /
      private Connection psconn;
      /
      *
    • to query the PG db
      /
      private Statement stmt;
      private static PreparedStatement orgidStmt;
      /
      *
    • loaded in constructor
      /
      private Vector unusedIds;
      public static final String queryCDS = "select s, s->corresponding_protein[1], len, ss->Species[1] from s in class sequence, len in (s->CDS[2]-s->CDS[1]+1), ss in s->Source where exists_tag s->CDS and len > 18 and not exists_tag s->cDNA_EST";
      public static final String querymRNA = "select s, s->corresponding_protein[1], len, s->Species[1] from s in class Sequence, len in (s->CDS[2]-s->CDS[1]+1) where exists_tag s->CDS and len > 18 and exists_tag s->cDNA_EST";
      /
      *
    • unused id index
      /
      private int unusedIndex=0;
      /
      *
    • for keeping track of the ids
      /
      private static int cds_id=0;
      /
      *
    • set a limit on ace names
      */
      private static final int ACE_ID_LEN = 34;

    public LoadCDSFromAce(String pgDB, String usr)
    throws SQLException, ClassNotFoundException
    {
    user = usr;
    this.pgDB = pgDB;
    Class.forName("org.postgresql.Driver");
    psconn = DriverManager.getConnection(pgDB, usr, pgPassword);
    stmt = psconn.createStatement();
    orgidStmt = psconn.prepareStatement("Select id from species where name=?");
    unusedIds=getUnusedId();
    }

    /**

    • no doc yet
      */
      public LoadCDSFromAce() throws SQLException, ClassNotFoundException {
      Class.forName("org.postgresql.Driver");
      psconn = DriverManager.getConnection(pgDB, user, pgPassword);
      stmt = psconn.createStatement();
      orgidStmt = psconn.prepareStatement("Select id from species where name=?");
      unusedIds=getUnusedId();
      }

    /**

    • This is to connect to an ACEDB on host h and port p.
      */
      public void connect(String h, int p) {
      aceDB = new Ace(h, p, user, acePassword);
      }

    public void connectAce(String dbname) {
    aceDB = createAce(dbname);
    }

    public void close() throws SQLException {
    orgidStmt.close();
    psconn.close();
    }
    public void closeAce() { aceDB.close(); }

    /**

    • Update the cds table from ACEDB, updates every existing ace_id
    • and inserting new ace_id; more efficient if cds table is already loaded.
      */
      public void updateFrom(String aceQuery, boolean mRNA)
      throws SQLException, AceException {
      Result seqSet = aceDB.execQuery(aceQuery);
      PreparedStatement insertStmt= psconn.prepareStatement(
      "INSERT INTO cds(id, ace, prt, length, source, from_mRNA) VALUES(?, ?, ?, ?, ?, ?)");
      insertStmt.setBoolean(6, mRNA); // once for each load
      PreparedStatement updateStmt= psconn.prepareStatement(
      "UPDATE cds set length=?, source=?, from_mRNA=? where ace=?");
      updateStmt.setBoolean(3, mRNA);

    String aceid = "null", prtid="null";
    String organism="";
    int updateCnt=0, insertCnt=0, skipped=0, length, source;
    while (seqSet.next()) {
    try {
    aceid= seqSet.getString(1);
    if (aceid.length() > ACE_ID_LEN) {
    System.err.println(aceid + "| longer than " + ACE_ID_LEN);
    }
    prtid=seqSet.getString(2);
    length=seqSet.getInt(3);
    source = getOrgid(seqSet.getString(4));

         updateStmt.setString(4, aceid);
         updateStmt.setInt(1, length);
         updateStmt.setInt(2, source);
         if (updateStmt.executeUpdate() == 0) {
            //System.out.println("Cannot update "+aceid+" | "+
            //      length+" | "+source);
            insertCnt++;
            insertStmt.setInt(1, ((Integer)unusedIds.get(unusedIndex++)).intValue());
            insertStmt.setString(2, aceid); 
            if (prtid.equals("")) insertStmt.setNull(3, Types.VARCHAR);
            else insertStmt.setString(3, prtid); 
            insertStmt.setInt(4, length);  
            insertStmt.setInt(5, source); 
            insertStmt.executeUpdate();
         }
         else updateCnt++;
      }
      catch (SQLException se) {
         if (se.toString().indexOf("Cannot insert a duplicate key into unique index cds_ace_key") != -1) {
            System.err.println("cannot insert ace");
            System.exit(1);
            skipped++;
         }
         else {
            System.err.println(insertStmt);
            for (int i=1; i<=seqSet.getColumnCount(); i++) 
               System.err.print(seqSet.getString(i) + " | ");
            System.out.println("\n");
            System.out.println(se);
            se.printStackTrace();
            System.exit(1);
         }
      }
    

    }
    System.out.println("Insert cds attempts " + seqSet.getRowCount());
    System.out.println("Total updates " + updateCnt + " total inserts " + insertCnt);
    insertStmt.close();
    updateStmt.close();
    }

    /**

    • use this function when the cds table is originally empty,

    • Or all of the entries are mostly new.

    • When called for both CDS and CDS_from_mRNA, usually the second call fails

    • with large databases such as bony, human, etc.

    • I have no solutions yet. The failer is from the backend.
      */
      public void loadFrom(String aceQuery, boolean mRNA)
      throws SQLException, AceException {
      //System.out.println("Executing the ace query " + aceQuery + " ... ");
      Result seqSet = aceDB.execQuery(aceQuery);
      PreparedStatement insertStmt= psconn.prepareStatement(
      "INSERT INTO cds(id, ace, prt, length, source, from_mRNA) VALUES(?,?,?,?,?,?)");
      insertStmt.setBoolean(6, mRNA); // once for each load
      String aceid = "", prtid="null";
      String organism="";
      int seqcnt=0;
      int skipped=0;

      while (seqSet.next()) {
      //System.out.println("sequence: " + seqcnt + " skip: " + skipped); // for bugy machines ..
      if (seqcnt%1000 == 0) System.out.print(".");
      try {
      seqcnt++;
      cds_id = nextID(); // constrols its own id should not get duplicate keys
      insertStmt.setInt(1, cds_id);
      aceid= seqSet.getString(1);
      prtid= seqSet.getString(2);
      insertStmt.setString(2, aceid);
      if (prtid.equals("")) insertStmt.setNull(3, Types.VARCHAR);
      else insertStmt.setString(3, prtid);
      insertStmt.setInt(4, seqSet.getInt(3));

        organism=seqSet.getString(4);  // organism may be empty, this happened with the fugu
        // database, where I loaded a lot of scaffold sequence with giving them the source
        if ( organism.equals("")) {
           String tmp = aceDB.getDBInfo();
           //String tmp1 = aceDB.getDBTitle();
           //System.err.println("info: " + tmp + " title: " + tmp1);
           if (tmp.indexOf("Fugu") != -1) { organism = "Fugu rubripes"; }
           else { System.exit(1); }
        }
        //     insertStmt.setInt(5, getOrgid(seqSet.getString(4)));  // orgname could be null!
         insertStmt.setInt(5, getOrgid(organism));  // orgname could be null!
        insertStmt.executeUpdate();
      

      }
      catch (SQLException se) {
      if (se.toString().indexOf("Cannot insert a duplicate key into unique index cds_ace_key") != -1) {
      //System.out.println("ace_id already in db: " + aceid);
      skipped++;
      }
      else {
      System.out.println("id=" + cds_id);
      for (int i=1; i<=seqSet.getColumnCount(); i++)
      System.err.print(seqSet.getString(i) + " | ");
      System.out.println("\n");
      System.out.println(se);
      se.printStackTrace();
      System.exit(1);
      }
      }
      }
      seqSet.close();
      System.out.println("Insert cds attempts " + seqSet.getRowCount());
      System.out.println("Total skipped " + skipped);
      }

    /**

    • create a file for the postgres \Copy command, this can speed up

    • Database loading by 1000x. Duging loading also drop index. This

    • can also greately increase the speed of loading
      */
      public void createFileFrom(String aceQuery, boolean mRNA)
      throws SQLException, AceException {
      Result seqSet = aceDB.execQuery(aceQuery);
      //PreparedStatement insertStmt= psconn.prepareStatement(
      // "INSERT INTO cds(id, ace, prt, length, source, from_mRNA) VALUES(?,?,?,?,?,?)");
      //insertStmt.setBoolean(6, mRNA); // once for each load
      String aceid = "", prtid="null";
      String length;
      String p_cluster = "\N";
      String n_cluster = "\N";
      String orgid; // same as source;
      String from_mrna;
      if (mRNA) from_mrna = "t";
      else from_mrna = "f";
      String organism="";
      int seqcnt=0;
      int skipped=0;
      while (seqSet.next()) {
      if (seqcnt%1000 == 0) System.out.print(".");
      try {
      seqcnt++;
      cds_id = nextID(); // constrols its own id should not get duplicate keys
      aceid= seqSet.getString(1);
      prtid= seqSet.getString(2);
      length = seqSet.getString(3);
      if (prtid.equals("")) prtid = "\N";

        organism=seqSet.getString(4);  // organism may be empty, this happened with the fugu
        if ( organism.equals("")) {
           String tmp = aceDB.getDBInfo();
           if (tmp.indexOf("Fugu") != -1) { organism = "Fugu rubripes"; }
           else { System.exit(1); }
        }
        orgid = getOrgidString(organism);
        System.out.println(cds_id + " " + aceid + " " + prtid + " " + length + " " + p_cluster + " " + n_cluster + " " + orgid + " " + from_mrna);
      

      }
      catch (SQLException se) {
      System.out.println("id=" + cds_id);
      for (int i=1; i<=seqSet.getColumnCount(); i++)
      System.err.print(seqSet.getString(i) + " | ");
      System.out.println("\n");
      System.out.println(se);
      se.printStackTrace();
      System.exit(1);
      }
      }
      seqSet.close();
      System.out.println("Insert cds attempts " + seqSet.getRowCount());
      System.out.println("Total skipped " + skipped);
      }

    public void insert() throws SQLException, AceException {
    System.err.println("updating CDS ... ");
    // the problem is that client time out too short for long insertions
    loadFrom(queryCDS, false); // only when this line is commented out the
    // human loading can proceed
    System.err.println("updating CDS from mRNA ... ");
    //loadFrom(querymRNA, true);
    }

    /**

    • Load all organisms from this acedb to the species table in the

    • relational database if organism already exist then just ignore.

    • So this function can update an existing organism table

    • div provides a division such as rod, vrt

    • depends on postgres sequence to keep track of the id.

    • @param div NCBI division
      */
      public void loadOrganism(String div) throws SQLException,
      AceException {
      String orgQuery = "select s->species from s in class sequence";
      Result orgSet = aceDB.execQuery(orgQuery);
      PreparedStatement pstmt =
      psconn.prepareStatement("Insert into species(name, db) values(?, '" + div + "')");
      String thisOrg = "";
      int count=0;
      int skipped =0;
      while (orgSet.next()) {
      try {
      count++;
      thisOrg = orgSet.getString(1);
      if (thisOrg.length() > 80) { // organism name length max 80 char
      System.err.println(thisOrg + "|| Name longer than 80");
      }
      pstmt.setString(1, thisOrg);
      pstmt.executeUpdate();
      }
      catch (SQLException se) {
      if (se.toString().indexOf("Cannot insert a duplicate key into unique index species_name_key") != -1) {

           System.err.println("Species: " +
                 thisOrg + " is already in the postgres DB. " +
                 "We skip this one.");
        
           skipped++;
        }
        else {
           System.err.println("Error in adding species. "+ se);
           System.exit(1);
        }
      

      }
      }
      pstmt.close();
      System.err.println(count + " Insert organisms attempted\n");
      System.err.println(skipped + " skipped\n");
      }

    /**

    • does a database query from species table to obtain the id of species
    • from name, orgName must be scientific.
      */
      public int getOrgid(String orgName) throws SQLException {
      orgidStmt.setString(1, orgName);
      ResultSet rs = orgidStmt.executeQuery();
      if (!rs.next()) {
      System.err.println(orgName + " not in database " + pgDB);
      System.exit(1);
      }
      return rs.getInt(1);
      }

    public String getOrgidString(String orgName) throws SQLException {
    orgidStmt.setString(1, orgName);
    ResultSet rs = orgidStmt.executeQuery();
    if (!rs.next()) {
    System.err.println(orgName + " not in database " + pgDB);
    System.exit(1);
    }
    return rs.getString(1);
    }

    /**

    • To obtain unused cds_id in the cds table, run at
    • object construction and called by the constructor.
      */
      private Vector getUnusedId() throws SQLException {
      Vector ids = new Vector(130000);
      ResultSet rs = stmt.executeQuery(
      "Select id from cds order by id");
      int cnt=0;
      while (rs.next()) {
      cnt++;
      while (rs.getInt(1) > cnt) {
      ids.add(new Integer(cnt++));
      }
      }
      rs.close();
      System.err.println(ids.size() + " unused ids");
      if (ids.size() == 0) { // no more free unused ids
      rs = stmt.executeQuery("Select max(id) from cds");
      rs.next();
      cds_id=rs.getInt(1); // always the largest id in the cds table
      }
      return ids;
      }

    private int nextID() {
    if (unusedIndex < unusedIds.size())
    return ((Integer)unusedIds.get(unusedIndex++)).intValue();
    else {
    unusedIds.clear();
    return ++cds_id;
    }
    }

    public static void main(String[] args) {
    final String divs[]= { "amp", "bony", "cart", "fugu", "human", "mam", "mouse", "plc", "pri", "rod", "sau", "vrt"};
    int i = 0, startIdx=0;
    boolean org = false, loadDivOnly = false;
    String div=""; //orgName="";
    while (i<args.length) {
    if (args[i].equals("-o")) org = true; // load org only
    else if (args[i].equals("-d")) div = args[++i];
    //else if (args[i].equals("-s")) orgName = args[++i];
    else if (args[i].equals("--div")) loadDivOnly = true;
    else if (args[i].equals("-b")) startIdx = Integer.parseInt(args[++i]); // for reload from failed operation
    i++;
    }

    try {
    LoadCDSFromAce lc = new LoadCDSFromAce();
    if (!div.equals("")) { // loading from specific database
    lc.connect( (String)acename2host.get(div), ((Integer)acename2port.get(div)).intValue());
    if (org) lc.loadOrganism(div);
    else lc.insert();
    lc.closeAce();
    }
    else {
    // load from all databases
    for (i=startIdx; i<divs.length; i++) {
    System.out.println("\nWorking on " + divs[i]);
    lc.connect((String)acename2host.get(divs[i]), ((Integer)acename2port.get(divs[i])).intValue());
    if (org) lc.loadOrganism(divs[i]);
    else lc.insert();
    lc.closeAce();
    }
    if (loadDivOnly) {
    lc.close();
    return;
    }
    }
    lc.close();
    }
    catch (AceException ae) {
    System.err.println("Ace error " + ae);
    ae.printStackTrace();
    }
    catch (SQLException sqle) {
    System.err.println(sqle);
    sqle.printStackTrace();
    }
    catch (ClassNotFoundException ce) {
    System.out.println(ce);
    ce.printStackTrace();
    System.exit(1);
    }
    }
    }

let me know you need more. It crashed in the lexer part I am not an expert on lex. If it was pure C++ or C I could debug it for you.

On 2016-04-09 11:59:16 +0000, albert wrote:

I've just pushed a proposed patch to github (pull request 467)

On 2016-04-10 07:54:16 +0000, albert wrote:

Code has ben integrated in the main branch on github.

On 2016-09-05 13:45:35 +0000, Dimitri van Heesch wrote:

This bug was previously marked ASSIGNED, which means it should be fixed in
doxygen version 1.8.12. Please verify if this is indeed the case. Reopen the
bug if you think it is not fixed and please include any additional information
that you think can be relevant (preferably in the form of a self-contained example).

@doxygen doxygen closed this Jul 2, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment